Relay systems

ABSTRACT

Relay systems may be incorporated into optical systems to direct light from at least one image source to a viewing volume. Light from a plurality of image sources may be directed by relay systems to a viewing volume. Some light from the plurality of image sources may be occluded by an occlusion system to reduce undesirable artifacts in when the relayed light from the plurality of image sources are observed in the viewing volume.

TECHNICAL FIELD

This disclosure generally relates to systems configured for generatinglight corresponding to 2D, 3D, or holographic imagery and furtherconfigured to relay the generated holographic imagery to desiredlocations.

BACKGROUND

Many technologies exist today that are often confused with holograms butlack the ability to stimulate the human visual sensory response in thesame way that a real object does. These technologies include lenticularprinting, Pepper's Ghost, glasses-free stereoscopic displays,horizontal-only parallax displays, head-mounted VR and AR displays(HMD), and other such illusions generalized as “fauxlography.” Thesetechnologies may exhibit some of the desired properties of a trueholographic display, but they fall short of the ideal of a full-parallaxviewing experience with correct occlusion handling for any number ofviewers with no headgear or glasses required in which the light field isreproduced almost exactly as it exists when light emerges from a realobject.

SUMMARY

An embodiment of an optical system in accordance with the presentdisclosure may comprise a first input interface configured to receivelight along a first set of light paths from a first image source,wherein the light from the first image source is operable to define afirst image surface, a second input interface configured to receivelight along a second set of light paths from a second image source,wherein the light from the second image source is operable to define asecond image surface, and a relay system configured to direct thereceived light from the first and second image sources to a viewingvolume, wherein at least one of the first and second image surfaces isrelayed by the relay system into the viewing volume, wherein at leastone of the first and second image sources comprises a light fielddisplay, and the first set of light paths are determined according to afour-dimensional (4D) function defined by the light field display suchthat each light path from the light field display has a set of spatialcoordinates and angular coordinates in a first four-dimensionalcoordinate system.

An embodiment of an optical system in accordance with the presentdisclosure may comprise a first input interface configured to receivelight along a first set of light paths from a first image source,wherein the light from the first image source is operable to define afirst image surface, a second input interface configured to receivelight along a second set of light paths from a second image source,wherein the light from the second image source is operable to define asecond image surface, a relay system configured to direct the receivedlight from the first and second image sources to a viewing volume,wherein at least one of the first and second image surfaces is relayedby the relay system into the viewing volume, and an occlusion systemconfigured to occlude a portion of light from at least one of the firstand second image sources.

An embodiment of an optical system in accordance with the presentdisclosure may comprise an optical combining system comprising a firstinput interface configured to receive light along a first set of lightpaths from a first image source, wherein the light from the first imagesource is operable to define a first image surface, and a second inputinterface configured to receive light along a second set of light pathsfrom a second image source, wherein the light from the second imagesource is operable to define a second image surface, and a first relaysystem configured to receive combined image light from the opticalcombining system and relay the received light to relayed locations in aviewing volume thereby defining first and second relayed image surfacescorresponding to the first and second image surfaces respectively,wherein at least one of the first and second image sources comprises alight field display, and the first set of light paths are determinedaccording to a four-dimensional (4D) function defined by the light fielddisplay such that each light path from the light field display has a setof spatial coordinates and angular coordinates in a firstfour-dimensional coordinate system.

An embodiment of an optical system in accordance with the presentdisclosure may comprise an optical combining system comprising a firstinput interface configured to receive light along a first set of lightpaths from a first image source, wherein the light from the first imagesource is operable to define a first image surface, and a second inputinterface configured to receive light along a second set of light pathsfrom a second image source, wherein the light from the second imagesource is operable to define a second image surface, a relay systemconfigured to receive combined light from the optical combining systemand relay the received light to relayed locations in a viewing volume,whereby first and second relayed image surfaces are observable at therespective relayed locations, and an occlusion system configured toocclude a portion of light from at least one of the first and secondimage sources.

An embodiment of a display system in accordance with the presentdisclosure may comprise a relay system comprising at least onetransmissive reflector, first and second image sources operable tooutput light along first and second sets of source light paths,respectively, wherein the first and second image sources are orientedrelative to the at least one transmissive reflector such that lightalong the first and second sets of source light paths is relayed alongfirst and second sets of relayed light paths, respectively, the firstand second sets of relayed light paths defining first and second viewingvolumes, respectively, wherein the first and second relayed viewingvolumes are different.

An embodiment of a display system in accordance with the presentdisclosure may comprise a relay system comprising at least onetransmissive reflector, an image source operable to output light, and abeam splitter positioned to receive the light from the image source anddirect the light along first and second sets of source light paths,wherein the image source and beam splitter are oriented relative to theat least one transmissive reflector such that light along the first andsecond sets of source light paths is relayed along first and second setsof relayed light paths, respectively, the first and second sets ofrelayed light paths defining first and second relayed viewing volumes,respectively, and wherein the first and second relayed viewing volumesare different.

An embodiment of a relay system in accordance with the presentdisclosure may comprise a first relay subsystem comprising a firsttransmissive reflector of the first relay subsystem, the firsttransmissive reflector positioned to receive image light from an imagesource, the image light operable to define a first image surface,wherein the first transmissive reflector is configured to relay theimage light received along source light paths within first and secondranges of angular alignment relative to the first transmissive reflectorto define a first relayed image surface in a first relayed location, anda second transmissive reflector of the first relay subsystem, the secondtransmissive reflector positioned to receive relayed image light fromthe first transmissive reflector and configured to relay the relayedimage light from the first transmissive reflector to define a secondrelayed image surface in a second relayed location.

An embodiment of a display system in accordance with the presentdisclosure may comprise arrays of modular display devices, each modulardisplay device comprising a display area and a non-imaging area, whereinthe arrays of modular display devices define a plurality of displayplanes, each display plane comprising imaging regions defined by thedisplay areas of the respective display devices and non-imaging regionsdefined by the non-imaging areas of the respective display devices, alight combining system operable to combine light from the arrays ofmodular display devices, wherein the light combining system and thearrays of modular display devices are arranged such that the combinedlight has an effective display plane defined by superimposing theplurality of display planes so that the non-imaging regions of theplurality of display planes are superimposed by the imaging regions ofthe plurality of display planes.

An embodiment of a light field display system in accordance with thepresent disclosure may comprise arrays of modular display devices, eachmodular display device comprising a display area and a non-imaging area,wherein the arrays of modular display devices define a plurality ofdisplay planes, each display plane comprising imaging regions defined bythe display areas of the respective display devices and non-imagingregions defined by the non-imaging areas of the respective displaydevices, arrays of waveguides each positioned to receive light from theof the display plane of one of the arrays of modular display devices, alight combining system operable to combine light from the arrays ofwaveguides, wherein each array of waveguides is configured to directlight from the respective array of modular display devices such that thecombined light from the light combining system comprises light pathseach defined according to a four-dimensional function and having a setof spatial coordinates and angular coordinates in a firstfour-dimensional coordinate system.

An embodiment of an optical system in accordance with the presentdisclosure may comprise; a first input interface configured to receivelight along a first set of light paths from a first image source,wherein the light from the first image source is operable to define afirst image surface; a relay system configured to relay the receivedlight from the first image surface to a viewing volume to define arelayed first image surface, wherein the first image sources comprises alight field display, and the first set of light paths are determinedaccording to a four-dimensional (4D) function defined by the light fielddisplay such that each light path from the light field display has a setof spatial coordinates and angular coordinates in a firstfour-dimensional coordinate system, and; a sensor operable to collectdata related to a condition in the viewing volume.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an embodiment of a system configured to relay aholographic surface projected by a light field display using a beamsplitter and an image retroreflector;

FIG. 1B illustrates an embodiment of a system configured to relay aholographic surface projected by a light field display using a beamsplitter and a plurality of image retroreflectors;

FIG. 2A illustrates an embodiment of a corrective optical elementconfigured to reverse the polarity of U-V angular coordinates in afour-dimensional (4D) coordinate system;

FIG. 2B illustrates a top-level view of a waveguide placed over a numberof illumination source pixels in the U-V plane;

FIG. 2C illustrates a side view of the embodiment shown in FIG. 2B inthe U-Z plane with a thin lens as the waveguide;

FIG. 3A illustrates an embodiment of a holographic display systemsimilar to the system shown in FIG. 1A, in which the beam splitter andimage retroreflector have been replaced by a transmissive reflector;

FIG. 3B illustrates an embodiment of a holographic display system havingmultiple relay systems;

FIG. 3C illustrates another embodiment of a holographic display systemhaving multiple relay systems;

FIG. 4A illustrates a combined view of an embodiment of a dihedralcorner reflector array (DCRA);

FIG. 4B illustrates a side view of an embodiment of transmissivereflector imaging a point source of light;

FIG. 4C illustrates an embodiment of a holographic display system havinga relay system comprising a concave mirror;

FIG. 4D illustrates another embodiment of a holographic display systemhaving a relay system comprising a concave mirror;

FIG. 4E illustrates another embodiment of a holographic display systemhaving a relay system comprising a lens system;

FIG. 5A illustrates an embodiment of an ideal relay system;

FIG. 5B illustrates an embodiment of holographic display system having arelay system configured to relay first and second holographic surfacesprojected by a light field display using a beam splitter and an imageretroreflector;

FIG. 5C illustrates an embodiment of a holographic display system havinga relay system configured to relay first and second holographic surfacesprojected by a light field display using a beam splitter and a concavemirror;

FIG. 5D illustrates an embodiment of correcting the optical effect ofthe relay system shown in FIG. 5C;

FIG. 5E illustrates an embodiment of a holographic display system havinga relay system configured to relay first and second holographic surfacesprojected by a light field display using a beam splitter and a pluralityof concave mirrors;

FIG. 5F illustrates an embodiment of a holographic display system havinga relay system configured to relay first and second holographic surfacesprojected by a light field display using a beam splitter and a pluralityof reflective Fresnel mirrors;

FIG. 5G illustrates an ambient light rejection system using theconfiguration of FIG. 5F;

FIG. 5H illustrates the use of polarization controlling elements with anambient light rejection system;

FIG. 6 illustrates an embodiment of a holographic display system havinga relay system configured to relay first and second holographic surfacesprojected by a light field display using a transmissive reflector;

FIG. 7 illustrates an embodiment of a holographic display system havinga first relay system configured to relay first and second holographicsurfaces projected by a light field display and relay a third surfaceprojected by a second display;

FIG. 8A illustrates an embodiment of a holographic display system havinga second relay system, a plurality of displays, and an occlusion layer.

FIG. 8B illustrates an embodiment using the occlusion layer in FIG. 8Ato perform occlusion handling;

FIG. 8C illustrates an embodiment of a holographic display systemsimilar to that shown in FIG. 8A perceived by a viewer at a differentposition;

FIG. 9A illustrates an embodiment of a relay system having first andsecond relay subsystems;

FIG. 9B illustrates an operation of an occlusion system;

FIG. 9C illustrates another operation of an occlusion system;

FIG. 9D illustrates the effect of the occlusion system shown in FIG. 9Con the relayed real-world object image, as viewed by three observerpositions shown in FIG. 9A;

FIG. 9E illustrates an embodiment of a relay system comprised of tworelay subsystems comprising transmissive reflectors;

FIG. 9F illustrates the effect of the occlusion system shown in FIG. 9Eon the relayed real-world object image, as viewed by three observerpositions shown in FIG. 9E;

FIG. 9G illustrates an embodiment of a relay system having first andsecond relay subsystems with an additional input interface for lightfrom one or more image sources.

FIG. 9H illustrates an embodiment of a relay system having first andsecond relay subsystems;

FIG. 9I illustrates an alternative embodiment of the relay system shownin FIG. 9H;

FIG. 9J illustrates an alternative embodiment of the relay system shownin FIG. 9H;

FIG. 10A demonstrates the sequence of reflections and transmissions thatlight takes as it travels through an optical folding system;

FIG. 10B is a table tracking how light from a display changespolarization states after interacting with each layer of each path ofthe optical fold system of FIG. 10A;

FIG. 10C shows another embodiment of an optical folding system withselectable regions;

FIG. 10D is an orthogonal view of an optical fold system with increasedpath length for a selected region of light rays and an increased fieldof view;

FIG. 11A shows an embodiment of a relay system configured to relay lightfrom holographic object surfaces projected from a light field displaysimultaneously with the light from one or more real-world objects;

FIG. 11B illustrates an embodiment of a relay system that performs depthreversal;

FIG. 11C illustrates an embodiment of a relay system configured to relaylight from two image sources and reject ambient light;

FIG. 11D illustrates an embodiment of a relay system configured to relaylight from two sources;

FIG. 11E illustrates an embodiment of a relay system configured to relaylight from a display and one other source.

FIG. 11F illustrates another embodiment of a relay system configured torelay light projected from a first image source simultaneously with thelight from a second image source;

FIG. 11G illustrates an embodiment of a relay system configured to relaylight projected from a first image source and simultaneously transmitlight from a second image source.

FIG. 11H illustrates yet another embodiment of a relay system with twointerfaces configured to relay light from two image sources.

FIG. 11I illustrates an embodiment of a relay system configured to relaylight projected from a first image source comprising a real-world objectsimultaneously with the light from a second image source comprising areal-world object;

FIG. 11J illustrates an embodiment of a relay system configured to relaylight projected from a first image source and simultaneously transmitlight from a second image source.

FIG. 12 shows the configuration shown in FIG. 11A where the relay systemis realized by a transmissive reflector.

FIG. 13 shows the configuration shown in FIG. 12 , except that anoptical fold system has been placed between the light field display andthe beam splitter;

FIG. 14A shows the relay configuration shown in FIG. 13 , except that aninput relay system is included to relay the image of the real-worldobject;

FIG. 14B shows the relay configuration shown in FIG. 12 , except that aninput relay system is included to relay the image of a real-world objectto a location on the opposite side of the transmissive reflector fromthe viewer

FIG. 15 shows an embodiment of a relay system comprised of a beamsplitter and one or more retroreflectors;

FIG. 16 shows an embodiment of a relay system comprised of a beamsplitter and a single retroreflector;

FIG. 17 shows an embodiment of a relay system comprised of a beamsplitter and more than one concave mirrors.

FIG. 18 shows an embodiment of a relay system comprised of a beamsplitter and two Fresnel mirrors.

FIG. 19 shows an embodiment of a relay system comprised of a beamsplitter and a single Fresnel mirror;

FIG. 20 shows an example of an in-line relay system;

FIG. 21A shows holographic objects projected from a light field displayand viewed by an observer;

FIG. 21B shows the projection of holographic objects obtained when theu-v angular light field coordinates in FIG. 21B have been reversed;

FIG. 21C shows how the holographic objects shown in FIG. 21B are relayedwith the relay system shown in FIG. 20 ;

FIG. 22 shows a relay system comprised of an in-line relay system and anoptical fold system;

FIG. 23 shows the relay configuration of FIG. 22 but with the real-worldobject replaced by an input relay system.

FIG. 24 shows a configuration for a relay system comprised of one ormore lenses;

FIG. 25A illustrates an orthogonal view of a relay system in which thelight from at least one object is relayed by passing through the samerelay twice by reflecting from one or more mirrors;

FIG. 25B illustrates orthogonal views of a relay system in which thelight paths from at least one object are received and relayed by passingthe light rays through a transmissive reflector relay a first time,reflecting from a mirror, and passing the reflected light rays throughthe same relay a second time;

FIG. 25C illustrates a partial view of a relay system comprised of amirrored surface disposed at an angle to a transmissive reflector;

FIG. 25D illustrates more light paths for the relay in FIG. 25C;

FIG. 25E illustrates light paths being received and relayed by the relayof FIG. 25C;

FIG. 26A shows the coordinated movement between a holographic object andan occlusion region on an occlusion plane within a display system with arelay;

FIG. 26B shows the coordinated movement between a holographic object andan occlusion object within a display system with a relay;

FIG. 26C shows the movement of three relayed images and an occlusionregion of an occlusion plane when a relay within a display system isphysically moved;

FIG. 26D shows options for motorized movement of some of the componentsof the relay system shown in FIG. 26A;

FIG. 27A shows a combined field-of-views for two relays which is largerthan the field-of-view for either of the relays separately;

FIG. 27B shows two relays shown in FIG. 14 placed together to result ina larger combined field-of-view;

FIG. 27C shows the combined relay system of FIG. 27B after theadjustments have been made to have a larger combined field-of-view thaneither of the separate relays;

FIG. 27D shows two relays comprised of concave mirrors and beamsplitters arranged to achieve a larger field-of-view;

FIG. 27E shows two inline relays arranged to achieve a largerfield-of-view;

FIG. 27F shows two relays shown in FIG. 9G placed together to allowalmost twice the field-of-view of the separate relays;

FIG. 27G is a top view of a display system comprised of three separaterelays forming a single combined field of view;

FIG. 27H is a side view of FIG. 27G;

FIG. 27I shows the light from a holographic object being relayed andcombined with other light within a portion of the display system of FIG.27G;

FIG. 27J shows the light from a real-world object being relayed andcombined with other light within a portion of the display system of FIG.27G;

FIG. 27K shows the light from a real-world object being combined withother light within a portion of the display system of FIG. 27G;

FIG. 27L shows light from a display being combined with other lightwithin a portion of the display system of FIG. 27G;

FIG. 27M shows a front view of the display surface of the display systemof FIG. 27G;

FIG. 27N shows an off center view of the display surface of the displaysystem of FIG. 27G;

FIG. 27O shows a relay configuration comprised of two paralleltransmissive reflectors wherein only light incident at an acute angle tothe surface of the first transmissive reflector is relayed effectively;

FIG. 27P is a side view of the relay system shown in FIG. 27O with anadditional optical path for light which is at a normal angle to thesurface of the first transmissive reflector.

FIG. 28A illustrates a table-top display system comprised of an imagesource, a beam splitter, and a transmissive reflector;

FIG. 28B shows the display system of FIG. 28A with an additionalinterface for another image source;

FIG. 28C shows the display system of FIG. 28B with an occlusion planeand an additional relay;

FIG. 28D shows a table-top display system comprised of two image sourcesand a transmissive reflector;

FIG. 28E shows a table-top display system comprised of four imagesources and a transmissive reflector;

FIG. 28F shows a table-top display system which supports foregroundrelayed surfaces occluding background relayed surfaces;

FIG. 29A shows a top view of two display devices with each displaycomprised of a display area and a non-imaging area;

FIG. 29B shows a side view and an end view of the display device shownin FIG. 29A;

FIG. 29C shows multiple displays placed on a first plane A, and multipledisplays placed on a second plane B;

FIG. 29D shows a side view of first display plane A and second displayplane B of displays disposed orthogonal to one another;

FIG. 29E shows the combined light of FIG. 29D as viewed by the observer,with display plane A and display plane B superimposed;

FIG. 29F shows two display planes of display devices placed on a regularrectangular grid;

FIG. 29G shows a combined image of the display planes A and B shown inFIG. 29C, where the display plane A is rotated 90 degrees relative tothe other display plane B;

FIG. 29H shows a display plane C comprised of a regular rectilinear gridof display devices placed size-by-side in rows;

FIG. 29I shows a side view of one embodiment of a light combining systemcomprising two optical combiners combining the light from three displayplanes;

FIG. 29J is the combined light observed by an observer of the threedisplay planes shown in FIG. 29I;

FIG. 29K shows an embodiment in which each pixel is comprised of threerectangular subpixels;

FIG. 29L shows four identical display planes, display plane I, displayplane J, display plane K, and display plane L, each comprised of apattern of displays with spaces between each display and its neighbors;

FIG. 29M shows four display planes I, J, K, and L as shown in FIG. 29Lcombined using three optical combiners to form a display system;

FIG. 29N shows overlapping display planes from the configuration shownin FIG. 29M, with an effective overlapped seamless 2D display surface;

FIG. 29O shows the configuration of four overlapping display planes I,J, K, and L that produce the combined light I+J+K+L seen by an observerfrom the configuration shown in FIG. 29M;

FIG. 30A shows a waveguide system placed over an illumination plane,which is comprised of individually addressable pixels located on aseamless display surface;

FIG. 30B shows a light field system comprised of an array of waveguidesover pixels on an illumination plane which forms a seamless displaysurface;

FIG. 30C shows a side view of a light field display comprised of thedisplay device shown in FIG. 29B with a waveguide array shown in FIG.30B mounted onto its active display area surface;

FIG. 30D shows a magnified view of a portion of a display device with anactive display area covered with an array of waveguides, surrounded by anon-imaging area;

FIG. 30E shows two holographic objects projected by alight field displaysystem comprised of five waveguides, each projecting light from a groupof associated pixels and perceived by an observer;

FIG. 30F shows the light field display shown in FIG. 30B, with a layerof smart glass placed in a plane parallel to the plane of waveguides anddisplaced a small distance from the surface of the waveguides;

FIG. 30G shows the light field display shown in FIG. 30F, where thevoltage source applies a sufficient voltage to the transparent smartglass electrodes for the smart glass to become transparent;

FIG. 31A shows a side view of an array of display devices, comprised ofindividual displays shown in FIGS. 29A and 29B;

FIG. 31B shows how a 2D array of display devices containing imaging gapsmay be combined with an array of energy relays to produce a seamlessdisplay system with a seamless display surface without non-imagingregions;

FIG. 31C shows an array of individual light field display units shown inFIGS. 30C and 30D;

FIG. 31D is one embodiment of a light field display that appears in manyof the diagrams of this disclosure;

FIG. 32 shows a light field display comprised of an overlapped 2Ddisplay system formed from one or more planes of display devices, anoptical combiner, a relay system, and an array of waveguides placed at avirtual display plane;

FIG. 33 is a light field display similar to the light field displayshown in FIG. 32 , except that the two display planes in FIG. 32 arereplaced with a single seamless display surface, which may be anembodiment of the seamless display surface shown in FIG. 31B, and anoptional second seamless display surface;

FIG. 34A is a light field display system comprised of two arrays oflight field display devices, each of which may contain non-displayregions, combined by an optical combiner;

FIG. 34B shows how the display system shown in FIG. 34A appears to anobserver;

FIG. 34C shows the light field display system shown in FIG. 34A combinedwith a relay system which relays holographic objects to a virtualdisplay plane;

FIG. 35 shows a diagram of a display system shown in FIG. 11A wherein asensor records the gestures of a viewer and moves the relayed objects inresponse;

FIG. 36 shows the display system of FIG. 35 , with the path of lightfrom a viewer's hand travelling through the relay system in the oppositedirection from the direction of the combined light rays from the lightfield display and real-world object, with these reverse light raysdetected by a sensor.

DETAILED DESCRIPTION

FIG. 1A shows an embodiment of a holographic display system including afirst display 1001 comprising a light field display configured toproject light along a set of projected light paths 1036 to form at leasta first holographic surface 1016 having a first projected depth profilerelative to a display screen plane 1021. In an embodiment, the firstholographic surface 1016 may be any surface in a holographic scene, suchas a portion of an object, a face, a background scene, etc. In anembodiment, the projected depth profile of the holographic surface 1016may include a depth perceivable by a viewer (not shown) observing thefirst display 1001 along a normal axis (not shown) of the display 1001.The holographic display system of FIG. 1A also includes a relay system5010 positioned to receive light along the first set of projected lightpaths 1036 from the light field display 1001 and relay the receivedlight along a set of relayed light paths 1025A such that points on thefirst holographic surface 1016 are relayed to relayed locations therebyforming a first relayed holographic surface 1018 having a first relayeddepth profile relative to a virtual screen plane 1022. In an embodiment,the virtual screen plane 1022 is oriented at a non-parallel anglerelative to the display screen plane 1021 of the light field display1001. In an embodiment, the virtual screen plane 1022 is oriented at aperpendicular angle relative to the display screen plane 1021 of thelight field display 1001.

In an embodiment, the depth profile of the holographic surface 1016 mayinclude a depth perceivable by a viewer 1050 observing in the directionof the virtual screen plane 1022. As illustrated in FIG. 1A, the firstrelayed depth profile of the relayed holographic surface 1018 isdifferent from the first projected depth profile of the firstholographic surface 1016: first holographic surface 1016 is projected asan off-screen holographic surface while the first relayed holographicsurface 1018 is perceivable by viewer 1050 as an in-screen holographicsurface relative to the virtual screen plane 1022.

In an embodiment, the relay system 5010 may relay holographic objectsprojected by a light field display 1001 using a beam splitter 101 and animage retroreflector 1006A. In an embodiment, the light field display1001 comprises one or more display devices 1002, having a plurality oflight source locations (not shown), an imaging relay 1003 which may ormay not be present which acts to relay images from the display devicesto an energy surface 1005, and an array of waveguides 1004 which projecteach light source location on the energy surface 1005 into a uniquedirection (u,v) in three dimensional space. The energy surface 1005 maybe a seamless energy surface that has a combined resolution that isgreater than the surface of any individual display device of the one ormore display devices 1002. Examples of light field display 1001 aredescribed in commonly owned U.S. Pat. App. Pub. Nos. US2019/0064435,US2018/0356591, 2018/0372926, and U.S. patent application Ser. No.16/063,675, all of which are incorporated herein by reference for allpurpose. Projected light rays 1036 may converge at a location 113 on thesurface of a holographic object 1016, and then diverge as they approachthe beam splitter 101. The beam splitter 101 may be configured toinclude a polarizing beam splitter, a transparent aluminum-coated layer,or at least one dichroic filter. In an embodiment, the beam splitter 101may be oriented at a 45 degree angle relative to the light field displayscreen plane 1021 and the retroreflector 1006A, with the retroreflector1006A oriented orthogonally relative to the display screen plane 1021.Some fraction of the incident light along the projected light paths 1036reflects from the beam splitter 101 toward the image retroreflector1006A along a set of reflected light paths 1037, while some of theremaining light may pass straight through the beam splitter 101 intorays along a set of transmitted light paths 1039A, which may notcontribute to the formation of the relayed holographic object 1018 inthe configuration shown in FIG. 1A. In an embodiment, the retroreflector1006A may contain a fine array of individual reflectors, such as cornerreflectors. The retroreflector 1006A acts to reverse each ray ofincident light in the opposite direction from the approach direction,with no significant spatial offset. Rays along light paths 1037 reversetheir direction upon reflecting from the retroreflector 1006A,substantially retracing their approach angle to the retroreflector1006A, and some fraction of their intensities pass through the beamsplitter 101 along the set of relayed light paths 1025A, converging atthe location 114 of the holographic object 1018. In this way,holographic object 1016 projected directly by the light field display1001 is relayed to form the relayed holographic object 1018. Theretroreflector 1006A can be placed to the right of the beam splitter101, as shown in FIG. 1A, or placed above the beam splitter 101,orthogonal to the placement shown in FIG. 1A, directly facing the LFdisplay surface 1021 (in the same place as retroreflector 1006B shown inlater diagram FIG. 1B). In other words, the retroreflector can be placedso that light from LF display 1001 is reflected to the right by the beamsplitter, and then reflects from the retroreflector, or placed so thatlight from LF display 1001 is transmitted vertically by the beamsplitter, and then reflects from the retroreflector. Later in thisdisclosure, both orientations will be shown. In an embodiment, the lightfield display 1001 may include a controller 190 configured to issuedisplay instructions to the light field display and output lightaccording to a 4D function.

FIG. 1A may have an optional optical element 1041A located between thebeam splitter 101 and the retroreflector 1006A. The relative placementof this optional optical element 1041A is similar to the optionaloptical element 1041A that appears in FIG. 1B. This optical element maybe a polarization controlling element used together with a polarizationbeam splitter 101. If the display 1001 produces only one polarizationstate, then a polarizing beam splitter 101 may be arranged to directalmost all the light of the display toward the retroreflector 1006A,eliminating most of the light rays 1039A which may pass verticallythrough the beam splitter and not contribute to imaging the holographicobject 1018. Using a polarizing beam splitter 101, the light rays 1037are linearly polarized as they approach the optical element 1041A andare circularly polarized after passing through the optical element1041A, which may include a quarter wave retarder. Upon reflection fromthe retroreflector 1006A, most of the light on rays 1025A may becircularly polarized in the opposite direction, and for this oppositecircular polarization, the return pass through the quarter wave retarderwill result in these light rays converted to a linear polarization thatis rotated 90 degrees relative to the light rays 1027 approaching theretroreflector 1006A. This light has the opposite polarization to thelight that was reflected by the beam splitter 101, so it will passstraight through the beam splitter 101 rather than being deflected andcontribute to the imaging of holographic object 1018. In short, aquarter wave plate optical element 1041A placed between the beamsplitter 101 and the retroreflector 1006A may assist in converting themajority of light reflected from the beam splitter 101 from one linearpolarization to the opposite linear polarization, so that this light ispassed by the beam splitter 101 with optimal efficiency in generating aholographic image, and limited wasted light.

In cases where the display 1001 produces unpolarized light, about halfof the incident light 1036 on the beam splitter will be directed tolight rays along the set of light paths 1037 toward the retroreflector1006A, and about half of the incident light will be directed along a setof transmitted light paths 1039A, in the vertical direction. Thisresults in a loss of light rays 1039A. In an embodiment, as shown inFIG. 1B, the holographic display system of FIG. 1A may include a relaysystem 5020 that includes an additional retroreflector 1006B. In anembodiment, the additional retroreflector 1006B may be disposed oppositeto the display 1001 from the beam splitter 101, symmetric in distancebut orthogonal in orientation to retroreflector 1006A. FIG. 1B shows adisplay system which relays holographic surfaces projected by a lightfield display 1001 using a holographic relay system 5020 comprised of abeam splitter 101 and two image retroreflectors 1006A and 1006B, whereeach retroreflector reflects rays of incident light in the directionreverse of their incident direction. In FIG. 1B, the retroreflector1006A is labeled as optional, but the relay 5020 may operate withretroreflector 1006A present and retroreflector 1006B absent, withretroreflector 1006A absent and retroreflector 1006B present, or withboth retroreflectors 1006A and 1006B present. Both configurations may beimplemented in accordance with the principles of this disclosure. Incontrast to relay system 5010 in FIG. 1A in which the light rays alongthe transmitted paths 1039A are lost, in FIG. 1B the light rays alongthe transmitted paths 1039B are retroreflected from retroreflector 1006Bin the same way as rays along the reflected paths 1037 areretroreflected from retroreflector 1006A. Light rays along light paths1039B are reversed in direction by retroreflector 1006B and then reflectfrom the optical combiner 101 so that they are directed towards lightpaths 1025B which converge to form the holographic object 1018. Thelight rays along paths 1039B and paths 1037 are retroreflected andconverge at the beam splitter 101, combining to form light rays alongthe set of relayed paths 1025A and 1025B, wherein both sets of relayedlight paths 1025A and 1025B may focus at point 114, contributing to formthe first relayed holographic surface 1018. In an embodiment, theadditional retroreflector 1006B and the beam splitter 101 are alignedsuch that projected light that was transmitted through the beam splitter101 towards the additional retroreflector 1006B is reflected from theadditional retroreflector 1006B and further reflected by the beamsplitter 101 along an additional set of relayed light paths 1025Btowards the virtual display screen 1022, and the set of the relayedlight rays 1025A from first retroreflector 1006A and the additional setof relayed light rays 1025B from the additional retroreflector 1006Bsubstantially overlap. As discussed in regard to the optional opticalelement 1041A shown in FIG. 1A, the optical element 1041B may include aquarter wave retarder which may result in a majority of light rays alongthe transmitted paths 1039B returning to the beam splitter 101 with theopposite linear polarization, such that the majority of these light rayswill be directed by the beam splitter 101 toward the formation of theholographic surface 1018, rather than being transmitted straight throughthe beam splitter 101 and towards the display 1001. The optional opticalelement 1041B may contain polarization controlling elements, diffractiveelements, refractive elements, focusing or defocusing elements, or anyother optical elements.

Referring now to FIGS. 1A and 1B, in an embodiment, the verticaldistance D1 between location 113 on the directly projected surface 1016and the light field display screen plane 1021 may be the same as thehorizontal distance D1 between corresponding point 114 on the relayedholographic surface 1018 relative to the relayed virtual screen plane1022. The relay system 5010 or 5020 may be configured to relay aplurality of holographic surfaces distributed around light field displayscreen plane 1021, including the out-of-screen surface 1016 on the side1010 of the screen plane 1021, and surfaces that are projected in-screenon the side 1011 of the screen plane 1021. In the example shown in FIGS.1A and 1B, the surface 1016 is projected as an out-of-screen holographicsurface. These holographic surfaces may be relayed from screen plane1021 to virtual plane 1022 so that surfaces 1016 which are out-of-screenfor the screen plane 1021 appear behind the virtual plane 1022 withrespect to a viewer 1050, and similarly, so that surfaces that arein-screen for the light field display 1001, projected on the side 1011of screen plane 1021, appear in front of the virtual screen plane 1022with respect to a viewer 1050. For this reason, the depth of holographicsurface 1016 flips polarity—the location 113 of the out-of-screenholographic surface 1016 that is furthest away from the display screenplane 1021 is relayed to location 114 of the relayed holographic surface1018 that is furthest from the viewer 1050. To account for this reversalof depth, and to present the observer 1050 with the same view and samedepth profile of the relayed holographic surface 1016 that an observerof directly projected out-of-screen holographic object 1016 would seewithout the use of relay system 5020, the polarity of the U-V lightfield coordinates may be reversed. These U-V light field coordinates arethe two angular coordinates in the 4D light field function withcoordinates (X, Y, U, V). Reversing the polarity of the U-V light fieldcoordinates transforms projected light rays 1036 into projected lightrays 1013, each of which have the opposite slope. This convertsout-of-screen holographic projected surface 1016 into in-screenholographic projected surface 1014 with a reversed depth, which will berelayed into relayed holographic surface 1020. Relayed holographicsurface 1020 is out-of-screen relative to the virtual display plane 1022and will appear to observer 1050 to have the same depth profile relativeto the virtual screen plane 1022 as projected object 1016 has relativeto the display screen plane 1021. Projected holographic surface 1014will appear to be depth-reversed relative to the display screen plane1021. In summary, to project a holographic surface 1020 for observer1050 of the virtual screen plane 1022, the intended projectedholographic surface 1016 with the intended depth profile may be renderedfor the light field display 1021 without the effects of the relay 5010or 5020 being considered, and then each of the U-V angular light fieldcoordinates may be flipped to produce a depth-reversed surface 1014which appears on the opposite side of the display screen plane 1021 fromholographic object 1016, but which is relayed by relay system 5010 or5020 into relayed holographic object 1020 with the intended relayedholographic surface and the intended depth profile relative to thevirtual screen plane 1022. The 4D light field coordinate system for(X,Y,U,V) is described in in commonly-owned U.S. Pat. App. Pub. Nos.US2019/0064435, US2018/0356591, US2018/0372926, and U.S. patentapplication Ser. No. 16/063,675, which are incorporated herein byreference and will not be repeated here.

In an embodiment, each of the set of projected light paths 1036 has aset of positional coordinates and angular coordinates in afour-dimensional (4D) coordinate system defined with respect to thedisplay screen plane 1021, and each of the set of relayed light paths1025A, 1025B has a set of positional coordinates and angular coordinatesin a four-dimensional (4D) coordinate system defined with respect to thevirtual display plane 1022. As described above, holographic surface 1014may be rendered so that the light forming the surface of object 1014will be relayed as the intended distribution for the relayed surface1020, which may be directly viewed by observer 1050. One way to renderholographic surface 1014 is to first render holographic object 1016, theintended object to be shown in absence of relay systems 5010 or 5020,and then reverse in polarity its U-V angular coordinates. This reversalof U-V coordinates may result in holographic object 1014 being projectedinstead of object 1016, which may be relayed to the intended holographicobject 1020. The U-V polarity reversal may be done with a correctiveoptic element, as summarized below in reference to FIG. 2A, or using anadjustment in the 4D light field coordinates, possibly as a holographicobject rendering step, as summarized below in reference to FIGS. 2B and2C.

FIG. 2A shows an embodiment of a corrective optical element 20 whichacts to reverse the polarity of U-V angular light field coordinates. Twosubstantially identical planes 201, 202 of lenses are placed paralleland separated from one another. Each lens has a focal length f 200, andthe planes of lenses are oriented parallel to one another and separatedby a spacing of twice the focal length f 200, so that their focal planesoverlap at virtual plane 203, and so that lenses on opposite sides ofvirtual plane 203, such as 213 and 214, share a common optical axis 204.Incoming parallel light rays 211 are incident on lens 213 in plane 201with an incident angle to the optical axis 204 of ϑ in the U-Z plane,and φ in the V-Z plane. The light rays 211 are focused by lens 213 ontothe focal plane 203, and then diverge toward lens 214 which refracts therays into parallel rays 212. Parallel rays 212 leave lens 214 in plane202 with the reversed polarity angles of −ϑ with respect to the opticalaxis 204 in the U-Z plane, and −φ with respect to the optical axis 204in the V-Z plane, resulting in a direction that has been reversedrelative to the incident direction of parallel rays 211. This relaysystem may be placed above the screen plane 1021 in the path ofprojected light paths 1036 or in the relayed light paths 1025A, 1025Bshown in FIGS. 1A and 1B in order to reverse the polarity of U-Vcoordinates for projected holographic surfaces or relayed holographicsurfaces, respectively.

In an embodiment, the light field display 1001 may include a controller190, as shown in FIGS. 1A and 1B, configured to receive instructions foraccounting for the difference between the first projected depth profileand the first relayed depth profile by operating the light field display1001 to output projected light such that the first relayed depth profileof the first relayed holographic object is the depth profile intendedfor a viewer 1050. FIG. 2B shows a top-level view of a waveguide 221 ofthe light field display 1001 placed over a number of illumination sourcepixels 222 in the U-V plane, including a row of pixels at V=0, a columnof pixels at U=0, and individual pixels 223 and 224. In an embodiment,the waveguide 221 may be one of the waveguides 1004 in FIGS. 1A and 1B,and the pixels 222 may be on the energy surface 1005 in FIGS. 1A and 1B.In an embodiment, the waveguide 221 allows light from the pixels 222 tobe projected along the set of projected light paths where each projectedlight path has set of positional coordinates (X, Y) and angularcoordinates (U, V) in a four-dimensional (4D) coordinate system. Theprojected light paths may be light paths 1036 shown in FIGS. 1A and 1B.In order to reverse the polarity of the U-V coordinates and createholographic object 1014 from a light field rendered for holographicobject 1016 in FIGS. 1A and 1B, one would exchange the polarity of the Uand V coordinates as shown in the diagram, so that a pixel 224 with −Uand +V coordinates would swap places with a pixel 223 with +U and −Vcoordinates. All other pixels would swap positions as indicated by thedashed lines, with the exception of (U, V)=(0, 0) which stays in place.

FIG. 2C shows a side view of the embodiment shown in FIG. 2B in the U-Zplane with the waveguide 221 projecting the light from two differentpixel locations 223 and 224 on the pixel plane 222 along chief lightrays 232 and 231, respectively. The chief light rays 232 and 231 definethe axis of propagation for the light received from the correspondingtwo pixels and projected by waveguide 221, even if the light from eachpixel fills up a substantial portion of the aperture of the waveguide221. The two pixels 223 and 224 may be located at the minimum andmaximum U coordinates for a row of pixels 222 at a constant value of V.A reversal in the angular coordinate U may result in the chief light ray231 with angles 231A (ϑ, φ) relative to the optical axis 204 ofwaveguide 221 becoming chief light ray 232 which has the oppositeangular coordinates 232A (−ϑ, −φ) relative to the optical axis 204 butmay have the same intensity and color of the chief light ray 231. Ifsuch a reversal in angular light field coordinates (ϑ, φ), orequivalently (U, V) for each ray of a light field display then the depthprofile of a projected holographic object surface may be reversed, asshown above in reference to FIG. 1B.

FIG. 3A shows an embodiment of a holographic display system which issimilar to the configuration shown in FIG. 1A, except that the relaysystem 5010 shown in FIG. 1A comprised of the beam splitter 101 andimage retroreflector 1006A has been replaced by a relay system which iscomprised of a single transmissive reflector 5030 positioned to receivelight along the set of projected light paths 1036 from the light fielddisplay 1001 and direct the received light 1036 along the set of relayedlight paths 1026. In an embodiment, the transmissive reflector 5030internally reflects a portion of the received light 1036 among aplurality of internal reflective surfaces (described below in referenceto FIG. 4A) of the transmissive reflector 5030 and outputs light alongthe set of relayed light paths 1026 towards the virtual screen plane1022 in a first direction. Projected light rays 1036 from the lightfield display 1001 may converge at a location 113 on holographic surface1016, and then diverge as they approach the transmissive reflector 5030.The transmissive reflector 5030 internally reflects the diverging rays1036 such that they exit the other side of the reflector 5030 as raysalong the relayed paths 1026 and converge at location 114 of relayedholographic surface 1018. This may be accomplished within thetransmissive reflector 5030 through a sequence of multiple reflections,described in detail below. In this way, holographic surface 1016projected directly by the light field display 101 is relayed to formrelayed holographic surface 1018. In an embodiment, the display systemshown in FIG. 3A may include a controller 190 configured to issuedisplay instructions to the light field display and output lightaccording to a 4D function.

In an embodiment, the transmissive reflector 5030 is a dihedral cornerreflector array (DCRA). A first possible implementation of a DCRA is aplanar structure with numerous micromirrors placed perpendicular to thesurface of a substrate. The micromirrors may be square through holes,each hole providing internal walls which constitute small cornerreflectors. An incident light ray is reflected twice by two of theorthogonal adjacent internal walls of a square hole as the light raypasses through the DCRA, resulting in a retroreflection of the light rayin the plane of the structure while leaving the component of lightdirection perpendicular to the plane undisturbed. A second possibleimplementation of a DCRA is a structure with two thin layers ofclosely-spaced parallel mirror planes, oriented so the planes areorthogonal to one another as shown in FIG. 4A. In the embodimentillustrated in FIG. 4A, the transmissive reflector 5030 is constructedof two layers 406 and 407 of closely-spaced parallel reflective planeswherein the direction of the reflective planes 401 in layer 406 areoriented orthogonally to the direction of the reflective planes 402 inlayer 407 in a second dimension. Reflective surfaces 401 and 402 may bemirrored surfaces. In FIG. 4A, an incident light ray 404 that passesthrough the transmissive reflector is reflected a first time by a firstmirror 401 in the first plane of closely-spaced mirrors 406, andreflected a second time by a second mirror 402 in the second plane ofclosely-spaced mirrors 407, where mirror 401 and mirror 402 areorthogonal to one another. An incident light ray 404 reflects some ofits energy into reflected light ray 414 as it enters one side of theexternal surface 430 of the transmissive reflector. The amount ofreflection may be adjusted by adding an optical coating to one or bothsurfaces 430 of the transmissive reflector 5030. Light ray 404 has onecomponent of its momentum reversed upon the first reflective surface 401at location 410, and then has a substantially orthogonal component ofmomentum reversed upon a second reflection at point 411 from the secondreflective surface 402. The component of light ray 404 momentum in thedirection perpendicular to the surface 430 of the DCHA 5030 isunaffected.

FIG. 4B shows a side view of an embodiment of the operation of atransmissive reflector 5030, which may be the DCRA structure of dualthin parallel planes of mirrors just described in FIG. 4A, an array ofsquare through-holes arranged on a planar substrate described above, orsome other transmissive reflector. The transmissive reflector 5030 isshown imaging a point source of light 422 located a distance D fromtransmissive reflector 5030. The transmissive reflector 5030 is alignedparallel to the X-Y plane. Each of the rays of light 423 from the pointsource 422 has its X and Y momentum components reversed by transmissivereflector 5030, so that the light rays 424 that exit 5030 converge atimage point 425, a distance D from transmissive reflector 5030 but onthe opposite side of the transmissive reflector 5030 from source point422. In the embodiment described in FIGS. 4A and 4B, the redirection ofthe incident light rays 423 that occurs as a result of the tworeflections within the transmissive reflector 5030 causes thetransmissive reflector to act as a focusing element. A portion of thelight rays 423 reflect from one of the external surfaces 430 of thetransmissive reflector 5030, creating reflected light rays 433, and thefraction of reflected light may be controlled by applying an opticalcoating to the surface 430 of the transmissive reflector 5030.

Turning now to FIGS. 3B and 3C, it is possible to use a configurationwith more than one relay to relay holographic surfaces. If a holographicsurface is relayed twice, then the depth reversal of the holographicobject that may occur with the first relay may be undone with the secondrelay. This is generally true for holographic surfaces that are relayedby an even number of holographic relays. FIG. 3B shows a light fielddisplay system comprised of at least a first light field display 1001A,and two relay systems 130 and 140 which together relay at least a firstprojected holographic surface to a final relay location. In theembodiment shown in FIG. 3B, holographic surfaces 121A and 122A areprojected by light field display 1001A around the light field displayscreen plane 1021A and relayed to final relayed locations 121C and 122Caround a virtual display plane 1022B, with no depth reversal. Also shownin FIG. 3B is an optional second light field display 1001B, which mayproject an image surface 123A. In an embodiment, the display systemshown in FIG. 3B may include a controller 190 configured to issuedisplay instructions to the light field display 1001A and optional lightfield display 1001B and output light on each display according to arespective 4D function. In place of the second light field display1001B, the surface 123A may instead be the surface of a real-worldobject, or even the surface of a traditional 2D display. Light fromsurface 123A (whether it be the surface of a projected holographicobject, a real-world object, or a portion of a 2D display) will becombined with holographic surfaces 121A and 122A by the beam splitter101 and relayed by the pair of relay systems 130 and 140 to imageposition 123C, with no depth reversal. In the case that object 123A is areal-world object, then the holographic surfaces 121A, 122A and theimage of the real-world object 123A are combined and relayed together toholographic surfaces 121C, 122C, and 123C at relayed locations, allowingthe holographic surfaces and the real-world object to be displayedtogether free of a physical display plane.

In FIG. 3B, both relay systems 130 and 140 include transmissivereflectors 5030A and 5030B, respectively, but either one of these relayscould also be comprised of a beam splitter and a retroreflector likerelay 5010 shown in FIG. 1A. The holographic surfaces 121A and 122A areformed with light along a set of projected light paths 131A and 132Afrom light field display 1001A, respectively, and some fraction of lightalong the set of projected light paths are transmitted straight throughthe image combiner 101. The image combiner 101 may be any beam splitterdisclosed in the present disclosure. Projected light along the set ofprojected light paths 131A and 132A is relayed by first relay system 130along a first set of relayed light paths 131B and 132B which formdepth-reversed first and second relayed holographic surfaces 121B and122B, respectively, around first virtual screen plane 1022A. Light alongthe first set of relayed light paths 131B and 132B are relayed by thesecond relay system 140 along a second set of relayed light paths 131Cand 132C forming third and fourth relayed holographic surfaces 121C and122C, not depth-reversed, around a new virtual screen plane 1022B.Relayed holographic objects 121C and 122C should have the same depthprofile relative to screen plane 1022B as the depth profile of sourceprojected surfaces 121A and 122A relative to the screen plane 1021A,respectively.

Image surface 123A may be the surface of a real-world object, a portionof a 2D display surface, or a holographic surface projected by theoptional second light field display 1001B with a depth profile withrespect to the screen plane 1021B of the light field display 1001B. Inother embodiments, image surface 123A may be a relayed holographicobject. A portion of light 133Y from surface 123A is reflected by theimage combiner 101 into projected light paths 133A, while the otherportion passes through the image combiner 101 along a set of transmittedpaths 133Z. The transmissive reflector 5030A of relay system 130 hasreflective surfaces 430, and some of the incident light along theprojected paths 133A reflects into light paths 143A (and this is truefor light along the projected paths 131A and 132A, but this is not shownin FIG. 3B). A portion of light along light paths 133A from the surface123A are relayed by first relay system 130 to relayed light paths 133B,forming depth-reversed image 123B. A first portion of the light alongthe relayed light paths 133B reflect from the surface of transmissivereflector 5030B of relay system 140 along reflected paths 143B (this isalso true for incident light along relayed light paths 131B and 132B,but these reflections from the surface of transmissive reflector 5030Bare not shown FIG. 3B). The remaining portion of light along the relayedlight paths 133B are relayed a second time by second relay system 140 torelayed light paths 133C, forming relayed surface 123C, notdepth-reversed, which is either an image of a real-world object 123A, a2D image, or a relayed holographic surface 123A. For the case in whichsurface 123A is the surface of holographic object projected by lightfield display 1001B, relayed surface 123C has the same depth profile toobserver 1050 as the depth profile of surface 123A relative to screenplane 1021B, and first observer 1050 will see three relayed holographicsurfaces 121C, 122C, and 123C. For the case in which surface 123A is areal-world object, the relayed surface 123C has the same depth profileto observer 1050 as the real-world object, and first observer 1050 willsee the relayed holographic object alongside the relayed holographicsurfaces 121C and 122C. For the case in which surface 123A is a 2Ddisplay, first observer 1050 will see a relayed 2D display floating withrelayed holographic objects 121C and 122C.

In the display configuration shown in FIG. 3B with the second lightfield display 1001B in place, virtual screen plane 1022C is relayed fromthe corresponding second light field display screen plane 1021B, andthis virtual screen plane 1022C may be disposed a distance from virtualdisplay screen plane 1022B relayed from the first light field displayscreen plane 1021A. In this way the holographic content from the twolight field displays 1001A and 1001B may be superimposed into the samespace around virtual screens 1022B and 1022C, without depth reversal,allowing for an increase in the depth range for displaying holographicobjects that exceeds the depth range of either of the individual lightfield displays 1001A or 1001B. Note that each light field display 1001Aand 1001B may produce holographic objects in a holographic object volumein the neighborhood of corresponding display screen planes 1021A and1021B, respectively. The holographic object volume around display screen1021A is relayed to virtual screen plane 1022B, while the holographicobject volume around display screen plane 1021B is relayed to virtualscreen plane 1022C. The amount of separation between virtual screenplanes 1022B and 1022C is dependent on the difference in a firstdistance between display 1001A and the transmissive reflector 5030A, anda second effective optical distance between display 1001B and thetransmissive reflector 5030A. If these distances are the same, then thevirtual screen planes 1022B and 1022C will overlap. On the other hand,if the proximity of either light field display 1001A or 1001B from thetransmissive reflector 130 is adjusted, the relayed holographic objectvolumes in the neighborhood of the virtual screen planes 1022B and 1022Cmay be made to partially overlap to create a larger combined holographicobject volume, or be adjusted to create two distinct and separatedregions of relayed holographic object volumes appropriate for a givenapplication. In the event that the relayed holographic object volumesoverlap, then a combined relayed holographic object volume larger thanthe holographic object volume of either of the individual displays maybe achieved. Similarly, if a real-world surface 123A is used in place ofa projected holographic surface 123A, the relative positioning ofrelayed holographic objects 121C and 122C with the holographic image123C from the real-world object 123A may be adjusted and customized to aparticular application. Note that this discussion about variableseparation between virtual screen planes 1022B and 1022C can also beapplied to the case when only one relay is used, such as 130.

FIG. 3C is same display configuration shown in FIG. 3B but shows howlight that reflects from the second transmissive reflector 5030B of thesecond relay system 140 along reflected paths 141B, 142B, and 143B maybe received by a second observer 1051. The numbering in FIG. 3B appliesto FIG. 3C. Light along the first set of relayed light paths 131B and132B from depth-reversed relayed holographic objects 121B and 122B arereflected into reflected light paths 141B and 142B, respectively, andmay, in an embodiment, pass through a corrective optical element placedat plane 137. The corrective optical element may be similar to thatshown in FIG. 2A, acting to reverse the polarity of the angular lightfield coordinates (U, V), resulting in the second observer 1051perceiving the relayed holographic surfaces 121C and 122C with the samedepth profile relative to plane 137 as the depth profile of the sourceprojected surfaces 121A and 122A relative to display plane 1021 of lightfield display 1001A, respectively. In a similar way, the object 123A,which may be a holographic surface projected by display 1001B, or thesurface of a real-world object, produces rays of light which are relayedby relay system 130 along relayed light paths 133B, formingdepth-reversed image 123B, and a portion of these light rays 133B arereflected by the surface 430 of transmissive reflector 5030B into lightalong the reflected paths 143B. The optional corrective optical elementplaced at 137 just described may also reverse the depth so that secondobserver 1051 may see relayed image 123C with the same depth profile asthe depth profile of surface 123A. In this way observers 1050 and 1051will see the same holographic images in the same locations.

As previously described, if first observer 1050 sees depth-correctrelayed holographic images 121C, 122C, and 123C, then the correspondinglight along paths 141B, 142B, and 143B approaching plane 137 on its wayto second observer 1051 will be of depth-reversed images 121B, 122B, and123B. Instead of placing corrective optics at plane 137, it is possibleto instead use a third relay system (not shown) to reverse the depths ofthese depth-reversed images 121B, 122B, and 123B. An observer of thisthird relay (not shown) will see images relayed by the third relay atlocations different from the locations of holographic images 121C, 122C,and 123C perceived by the first observer 1050.

It is possible to use other focusing optical elements, defocusingoptical elements, mirrored surfaces, or any combination of these torelay a holographic object volume from a light field display. FIG. 4Cshows an embodiment of a display system in which a curved mirror is usedas a focusing element in place of a retroreflector to relay aholographic object volume without depth reversal. FIG. 4C shows anorthographic view of a display system with a holographic relay system5040 comprised of an optical combiner 462 and a concave mirror 452. Inan embodiment, the concave mirror 452 may be spherical, parabolic, orsome other shape. The optical combiner 462 may be any beam splitterdescribed herein. Since light produced along the vertical axis 454 willbe deflected by the optical combiner 462 into light along the opticalaxis 453 of the mirror 452, the vertical axis 454 is on the optical axisof the mirror 452, and so is a portion of object 461. In otherembodiments the object 461 may be displaced fully from the optical axis.The center of the curvature of the mirror C 451 is distance D1 away fromthe image combiner 462. The point C 451 is the relayed point of point C′441, which is also the same distance D1 away from the image combiner, onthe vertical optical axis 454. A portion of light leaving the point C′441 along a set of projected light paths 465 will reflect from the imagecombiner 462 along reflected light paths 466 incident on the mirror 452.The concave mirror 452 and the image combiner 462 are aligned such thatthe light rays 466 incident on the concave mirror 452 are reflected backthrough the image combiner 462 along a set of reflected light paths 467along a return direction substantially parallel but opposite indirection to the set of incident light paths 466. Light along thereflected light paths 467 may converge through point C 451 towards thevirtual screen plane 469. The object 461 may be a real-world object, orthe surface of a holographic object projected by a LF display 463.Similarly, light rays 471 from surface 461 will reflect from the imagecombiner 462 into reflected light paths 472 toward the concave mirror452. Light paths 472 in turn reflect from the concave mirror 452 andback through the image combiner 462 along light paths 474 whichcontribute to forming a relayed image 457 of the object 461 viewed byobserver 450. The optional optical layer 464 may containpolarization-controlling optics, lens elements, diffractive optics,refractive optics, or the like. In one embodiment, as described abovefor FIG. 3A, optical layer 464 is a quarter wave retarder which mayconvert linearly polarized light into circularly polarized light, andvice-versa. If a polarization beam splitter 462 is used, the lightleaving the beam splitter 462 on the reflected light paths 472 islinearly polarized in a first state. Rays along the light paths 472 maybe converted from this first state of linear polarization into a firststate of circular polarization incident on the mirror 452, which isconverted to a second state of circular polarization orthogonal to thefirst state upon reflection by the mirror 452, and further converted toa second state of linear polarization orthogonal to the first state oflinear polarization by the quarter wave retarder 464. The result islight rays 472 and light rays 474 have opposite states of linearpolarization so that almost all the light 471 first striking the opticalcombiner 462 may be directed to the mirror, and all the light 467approaching the optical combiner 462 after reflection from the mirrorwill pass through the polarization beam splitter 462 and contribute toimaging of the relayed object 457 viewed by viewer 450, rather thanbeing deflected. In the case of FIG. 4C where object 461 is aholographic surface projected by the LF display 463 around the displayscreen plane 468, the holographic object 461 is relayed to relayedholographic object 467 near corresponding relayed virtual screen plane469 and viewable by an observer 450. In an embodiment, surfaces in thevicinity of point C′ 441 are relayed into the vicinity of point C 451.

Another feature of the relay system of FIG. 4C is that objects that arecloser to the image combiner 462 than point C′ 441 are imaged to aposition further than the point C 451 from the image combiner, withmagnification, and objects that are further from the image combiner 462than point C′ 441 are imaged to a position closer than the point C 451from the image combiner 462, with minification. This means that thedepth ordering for holographic objects produced in the vicinity of pointC′ 441 is respected when they are relayed to point C 451. Themagnification or minification of objects in the vicinity of point C′ 441may be reduced by increasing the radius of curvature of mirror 452and/or making the depth range of the projected holographic objects smallabout point C′ 441 relative to the radius of curvature of the mirror452. While the example illustrated in FIG. 4B shows a spherical mirror,it is possible to use different configurations of mirrors to performimaging, including parabolic-shaped concave mirrors, and even convexmirrors which may be spherical or parabolic for projection of imageswith convergence points behind the mirror (to the right of the mirror452 in FIG. 4C), on the other side of the mirror from the viewer 450. Inan embodiment, the display system shown in FIG. 4C may include acontroller 190 configured to issue display instructions to the lightfield display 463 and output light according to a 4D function.

FIG. 4D is an orthogonal view of a display system with a holographicsurface 488 being relayed to holographic surface 489 using a holographicrelay system 5040 comprised of a curved concave mirror 482 and an imagecombiner 485, where the holographic surface is offset from the opticalaxis 483. The point 481 is a focal point of the mirror, which may bespherical, parabolic, or some other shape. As drawn, the surface 488 isa holographic surface projected from a light field display 497, but theimaging described here also works if the surface 488 is a real surface.Image combiner 485 may be any beam splitter discussed in thisdisclosure. Light paths 490C and 492C are projected at different anglesfrom the light field display 497 and converge to on a vertex of thesurface 488. These projected paths 490C and 492C reflect from the imagecombiner 485 (with some loss for light rays that pass directly throughthe image combiner, which is not shown) to become light rays alongreflected light paths 490D and 492D, which then reflect off the surfaceof the mirror 482 to become light rays on relayed paths 490E and 492E,respectively, which pass through the beam splitter (with some loss notshown) and converge again at one vertex of the image 489, helping formthe image 489. Light rays along paths 491C and 493C are projected atdifferent angles from the light field display 497 and converge to formanother vertex of the surface 488. These light rays along 491C and 493Creflect from the image combiner 485 (with some loss not shown) to becomelight rays along reflected paths 491D and 493D, which then reflect fromthe surface of the mirror 482 to become light rays on relayed paths 491Eand 493E, which pass through the image combiner 485 (with some loss, notshown) and converge again at one vertex of the image 489, helping formthe image 489. Light rays along projected paths 492C and 493C reflect aslight rays along reflected paths 492D and 493D from the image combiner,and pass through the focal point 481 of the curved mirror 482, turninginto rays along relayed paths 492E and 493E, which are parallel to theoptical axis 483. Light rays along projected paths 490C and 491C reflectfrom the beam splitter as light rays along reflected 490D and 491D,respectively, and are parallel to the optical axis before reflectingfrom the curved mirror 482, so their reflected rays along relayed paths490E and 491E, respectively, pass through the focal point 481 of thecurved mirror 482. In the configuration shown in FIG. 4D, holographicsurfaces projected by the LF display 497 around the screen plane 498,which may be the same as the display surface of the LF display 497, arerelayed to be projected around the virtual screen plane 469, viewable byan observer 450.

In an embodiment, light rays along projected paths 490C and 491C in FIG.4D are projected at a normal to the surface of the light field display497, at a single angle, or equivalently, a single value of light fieldangular coordinate, which we assign to be u=0 (u is in the plane of thedrawing—the orthogonal angular light field coordinate v is not discussedin reference to FIG. 4D, but similar comments apply to v as well). Theserays are reflected by the image combiner 485 into rays along reflectedpaths 490D and 491D, which then reflect from the mirror into rays alongthe relayed paths 490E and 491E. These two light rays, visible to theobserver 450, make different angles θ₁ and θ₂ with a normal 496 to aline 495 parallel with the virtual screen plane 496, and thus contributetwo different values of light field angular coordinate u to the imagingof the relayed holographic surface 489. In other words, despite bothrays having a single value of light field angular coordinate u=0 asprojected by the light field display 497, they have different values ofu at the relayed holographic surface 489, and this u value (orequivalently angle) is dependent in part on the position of the objectrelative to the focal point 481 of the mirror. Also, the two rays alongprojected paths 492C and 493C, projected at nonzero light field angularcoordinates from the light field display 497, reflect from the imagecombiner 485 and the mirror system to become light rays along relayedpaths 492E and 493E, both parallel to each other and parallel to anormal 496 to the virtual screen plane 469, so that they have the samelight field coordinate u=0 at this virtual screen plane 469, as viewedby the observer, despite being projected from the light field display497 with nonzero values of u. In other words, the angular light fieldcoordinates of the holographic surface 488 are rearranged by theholographic relay system 5040 comprised of the image combiner 485 andcurved mirror 482 in forming the relayed holographic surface 489. Tocorrect for this, the angular light field coordinates leaving the screenplane 498 of light field display 497 may be arranged in a compensatedmanner to achieve the desired angular light field coordinates leavingthe relayed virtual screen plane 469. Another perhaps unwanted effect isthat the normal to the light field display surface 498, usually thelight field angular coordinate u=0, often defines an axis of symmetryfor projected rays from the light field display surface 498. The lightrays produced at u=0 from the light field display 497, defining axes ofsymmetry from the light field display surface 498, may be relayed to thevirtual screen plane 469 with significant values of u (i.e. angle θ withthe normal 496 to the virtual screen plane 469 may vary), especially ifthe relayed holographic object 488 is offset significantly from theoptical axis 483. This may cause the field of view to be altered. Ingeneral, to minimize field-of-view changes for holographic surfacesrelayed by optical relay system shown in FIG. 4D, the light fielddisplay 497 may be centered close to the optical axis so thatholographic surfaces such as 488 may relayed to positions 489, alsoclose to the optical axis 483. In an embodiment, the display systemshown in FIG. 4D may include a controller 190 configured to issuedisplay instructions to the light field display 497 and output lightaccording to a 4D function.

In some embodiments, the focusing function of the mirror 482 shown inFIG. 4D may be replaced with one or more optical elements such aslenses, mirrors, or some combination of these elements. In oneembodiment of a display system, shown in FIG. 4E, the relay system 5040may be replaced by a relay system 5070 formed with one or more lenses.FIG. 4F shows an embodiment in which lens relay system 5070 comprised ofone or more lenses relays the holographic object 437 projected by thelight field display 463 to relayed holographic object 438. The one ormore lenses including lens 446 and optional lens 447 may have a commonoptical axis 454 that may be substantially aligned with a normal to thedisplay surface 468. The one or more lenses may perform a focusingfunction which optically relays the holographic object region around thelight field display screen plane 468 to a virtual screen plane 435 nearthe optical axis but on the far side of the one or more lenses from thelight field display 463. Light rays 486A, 487A projected from thesurface 468 of light field display 463 contribute to forming the 3Dsurface of holographic object 437, and these two light rays are relayedby lens 447 into light rays 486B, 487B which are then relayed into lightrays 486C, 487C by lens 446 to help form the relayed holographic surface438 viewed by observer 450. Optical systems with lenses may also containfocus points, resulting in magnification or minification of holographicobjects such as 437 as they are relayed. The relay 5070 may relay aprojected holographic object 437 that is in close proximity to aneffective focal length of the multiple lens 446, 447 system to a relayedlocation 438 which is at a greater distance from 5070, while relaying aprojected holographic object that is further to the right of 437 in FIG.4E to a relayed location which is at a lesser distance from 5070 to theright of 438 in FIG. 4E. In this case, the relay system 5070 may notreverse the depth profile of a projected holographic object 437, so therelayed surface 438 may have substantially the same depth profilerelative to virtual screen plane 435 as the depth profile of 437relative to the light field display 463 screen plane 468. In anembodiment, the display system shown in FIG. 4E may include a controller190 configured to issue display instructions to the light field display463 and output light according to a 4D function.

FIG. 5A shows an orthogonal view of a light field display systemcomprised of an ideal holographic object relay system 100 which relaystwo holographic objects projected on either side of a light fielddisplay screen plane 1021 at a first location and viewed to a firstobserver 1048, to two relayed holographic surfaces on either side of arelayed virtual display screen 1022 at a second location and viewed by asecond observer 1050. The light field display 1001 may output lightalong a set of projected light paths that includes light rays alongprojected light paths 1030Z that help form surface 1015Z in front 1010of light field display screen plane 1021, and light rays along projectedlight paths 1036Z that help form object 1016Z behind 1011 the screenplane 1021. Light paths 1035 are traced paths for the light rays 1036Zthat originate at the light field display surface, which in this exampleis collocated with the display screen plane 1021. Under idealcircumstances, the relayed holographic objects 1017A and 1018A on eitherside of virtual screen plane 1022 appear to observer 1050 exactly asdirectly projected holographic objects 1015Z and 1016Z appear toobserver 1048 in absence of any relay system 100. In other words, the LFdisplay 1001 and the relay system 100 should be configured so that lightrays along relayed paths 1032A and 1028A which form relayed holographicsurfaces 1017A and 1018A, respectively, reach observer 1050 in the sameway that the corresponding light rays along projected paths 1030Z and1036Z which form the directly projected holographic surfaces 1015Z and1016Z, respectively, reach observer 1048 in the absence of any relaysystem 100. From FIGS. 1A, 1B and 3A, and the discussion below, it willbe clear that to generate the relayed holographic objects 1032A and1028A using a practical implementation of a relay system 100, thelocation, depth profile, and magnification of projected objects 1015Zand 1016Z may have to be adjusted from their locations shown in FIG. 5A,and the light field angular coordinates may have to be rearranged foreach of these projected holographic source objects 1015Z and 1016Z. Inan embodiment, the display system shown in FIG. 5A may include acontroller 190 configured to issue display instructions to the lightfield display 1001 and output light according to a 4D function.

FIG. 5B shows an embodiment of a holographic display system similar tothe holographic display system of FIG. 1A. The holographic displaysystem of FIG. 5B includes a first display 1001, which may be a lightfield display configured to project light along a set of projected lightpaths 1030A and 1036A to form at least first and second holographicsurfaces 1015A and 1016A having first and second depth profiles relativeto a display screen plane 1021, respectively. The holographic displaysystem also includes a relay system 5010 positioned to receive lightalong the set of projected light paths 1030A and 1036A from the lightfield display 1001 and relay the received light along a set of relayedlight paths 1032A and 1028A such that points on the first and secondprojected holographic surfaces 1015A and 1016A are relayed to relayedlocations that form first and second relayed holographic surfaces 1017Aand 1018A, having first and second relayed depth profiles relative to avirtual screen plane 1022, respectively.

FIG. 5B shows a holographic relay system 5010 comprised of an imagecombiner 101 and an image retroreflector 1006A. The light field display1001 may be similar to the light field display 1001 discussed aboverespect to FIGS. 1A, 1B, 3A and 5A. The image combiner 101 may be a beamsplitter. The light field display 1001 projects out-of-screenholographic surface 1016A on the viewer side 1010 of the screen plane1021, and in-screen holographic surface 1015A on the display side 1011of the screen plane 1021. In an embodiment, the light field display 1001may output light along a set of projected light paths that includeslight rays along projected light paths 1036A that help form surface1016A, and light rays along projected light paths 1030A that help formin-screen surface 1015A (paths 1033 are ray trace lines that don'trepresent physical light rays). Each of the set of projected light paths1030A and 1036A has a set of positional coordinates (X,Y) and angularcoordinates (U,V) in a four-dimensional (4D) coordinate system definedby the light field display. These light rays may diverge as theyapproach the beam splitter 101. Some fraction of this incident light isreflected by the beam splitter 101 toward the image retroreflector 1006Aalong a set of reflected light paths that include paths 1037A from theincident light 1036A and paths 1031A from the incident light 1030A,while the remaining light 1034 not reflected by the beam splitter passesthrough the beam splitter and may be lost, not contributing to imagingof relayed holographic surfaces 1017A and 1018A. The retroreflector1006A may contain a fine array of individual reflectors, such as cornerreflectors. The retroreflector 1006A acts to reverse each ray ofincident light paths 1037A, 1031A in substantially the oppositedirection from the approach direction, with no significant spatialoffset. Light rays along reflected light paths 1037A reverse theirdirection upon reflecting from the retroreflector 1006A, substantiallyretrace their approach angle to retroreflector 1006A, and some fractionof their intensities pass through the beam splitter 101 along relayedlight paths 1028A, converging at the location 1018A of a holographicsurface. In this way, holographic surface 1016A projected directly bythe light field display 1001 is relayed to form relayed holographicsurface 1018A. Similarly, rays along light paths 1031A reverse theirdirection upon reflecting from the retroreflector 1006A, retrace theirapproach paths to retroreflector 1006A, and some fraction of theirintensities pass through the beam splitter along relayed light paths1032A, converging and forming holographic surface 1017A. In this way,holographic surface 1015A projected directly by the light field display1001 is relayed to form holographic surface 1017A. The relayed lightpaths 1028A and 1032A make up a set of relayed light paths thatoriginated from the set of projected light paths from the display 1001to the beam splitter 101 and then through the set of reflected lightpaths from the beam splitter 101 to the retroreflector 1006A, and backthrough the beam splitter 101. In an embodiment, each of the set ofrelayed light paths has a set of positional coordinates (X,Y) andangular coordinates (U,V) in a four-dimensional (4D) coordinate systemas defined by the relay system 5010. In-screen holographic surface1015A, which is projected at a greater depth than out-of-screen surface1016A by the light field display 1001, is relayed as surface 1017A,which is now closer to the viewer 1050 than surface 1018A relayed from1016A. In other words, the depth profile of holographic surfaces 1015Aand 1016A projected by the light field display is reversed by theholographic relay system 5010. The vertical distance between holographicsurface 1016A and the beam splitter 101 D1 is substantially the same asthe horizontal distance between the corresponding relayed holographicsurface 1018A and the beam splitter 101. Similarly, the verticaldistance D2 between holographic surface 1015A and the beam splitter 101is substantially the same as the horizontal distance D2 between therelayed surface 1017A and the beam splitter 101. As discussed withregard to the optional optical element 1041A shown in FIG. 1B, theoptical element 1041A in FIG. 5B is also an optional optical element.This 1041A may be a quarter wave retarder which may result in a majorityof light rays along paths 1031A or 1037A returning to the beam splitter101 with a linear polarization opposite from that of the light raysleaving the beam splitter 101, whereupon the majority of these lightrays will be directed toward the viewer 1050, rather than deflected bythe beam splitter 101 and towards the display 1001. Also, the light rayalong path 1042A of the projected light paths 1036A from holographicsurface 1016A, is projected from the light field display normal to thedisplay screen plane 1021, and usually is assigned to the angular lightfield coordinate value (u, v)=(0, 0). This light ray produces light rayalong relayed path 1042B, which helps form relayed holographic surface1018A. For observer 1050, the light ray 1042B is projected normal to thevirtual display plane 1022 and will be perceived as a ray with lightfield angular coordinate (u, v)=(0, 0) to observer 1050. To furthergeneralize, the optical relay system 5010 preserves the light ray atlight field coordinate (u, v)=(0, 0) to stay at that value, even afterbeing relayed, despite the required rearrangement of light field angularcoordinates that is shown in FIG. 2B to reverse depth with theretroreflector configuration shown in FIG. 5B. Alternatively, acorrective optical element may be included in the holographic displaysystem of FIG. 5B to reverse depth. In an embodiment, the correctiveoptical element 20 shown in FIG. 2A may be disposed in the set ofrelayed light paths 1028A and 1032A, the corrective optical element isconfigured to reverse the polarity of the angular coordinates (U,V) ofeach of the set of relayed light paths such that a viewer perceiving thefirst and second relayed holographic surfaces 1017A, 1018A through thecorrective optical element 20 will perceive a corrected depth order thatis the same as the depth order of the first and second holographicsurfaces 1015A, 1016A observed in absence of the relay 5010. In anembodiment, the corrective optical element may be disposed in thevirtual display plane. In another embodiment, a corrective opticalelement 20 may be disposed in the set of projected light paths 1030A,1036A and optically preceding the relay system 5010, and the correctiveoptical element 20 may be configured to reverse the polarity of theangular coordinates (U,V) of each of the set of projected light paths1030A, 1036A such that the first and second holographic surfaces 1015Aand 1016A have a reversed depth order. In an embodiment, the correctiveoptical element 20 may be disposed parallel to the display screen plane

FIG. 5C shows a light field display 1001 comprised of a relay system5040 similar to the relay system 5040 discussed above with respect toFIGS. 4C and 4D. In an embodiment, the holographic object volume relay5040 is comprised of an image combiner used to redirect diverging lightfrom holographic surfaces onto a concave reflective mirror 1007A whichrefocuses this diverging light into relayed holographic surfaces. Theimage combiner 101 may be a beam splitter. Retroreflector 1006A in FIG.5B has been replaced with a concave reflective mirror 1007A in FIG. 5C.The concave reflective mirror 1007A can be placed to the right of thebeam splitter 101, as shown in FIG. 5C, or placed above the beamsplitter 101, orthogonal to the placement shown in FIG. 5C, directlyfacing the LF display surface 1021 (in the same place as mirror 1007Bshown in later diagram FIG. 5E). In other words, the mirror can beplaced so that light from LF display 1001 is reflected by the beamsplitter, and reflects from the surface of the mirror, or placed so thatlight from LF display 1001 is transmitted by the beam splitter, andreflects from the surface of the mirror. Later in this disclosure, bothorientations will be shown. In the setup shown in FIG. 5C, in anembodiment, the mirror may be a spherical mirror with a radius ofcurvature approximately equal to the optical path length between thedisplay screen plane 1021 and the surface of the mirror, akin to themirror center of curvature C′ 441 in FIG. 4D being located at or nearthe screen plane 468 in FIG. 4C. The same holographic surfaces 1015A and1016A are projected by the light field display 1001 as shown in FIG. 5Balong a set of projected light paths 1030A, 1036A. The set of projectedlight paths 1030A and 1036A may be considered as determined according toa first four-dimensional (4D) function defined by the light fielddisplay 1001, such that each projected light path has a set ofpositional coordinates (X,Y) and angular coordinates (U,V) in a first 4Dcoordinate system defined with respect to a display screen plane 1021.Light 1030A from holographic surface 1015A reflects from the beamsplitter 101 into light rays along reflected light paths 1031A, andrather than being directed backwards along their same path as they werewith the retroreflector 1006A in FIG. 5B, these rays are reflected alongrelayed paths 1032B to converge and form holographic surface 1017B. Therelayed holographic surface 1017B is slightly smaller than the sourceholographic surface 1015A, due to minification performed by the concavemirror corresponding to the optical path length between holographicsurface 1015A and the mirror. In an embodiment, the mirror 1007A is aspherical mirror, and the optical path length between the holographicsurface 1015A and the mirror 1007A is slightly larger than the radius ofcurvature of the surface of mirror 1007A. Similarly, light 1036A fromholographic surface 1016A reflects from the beam splitter 101 into lightrays along reflected paths 1037A, and these rays are reflected alongrelayed paths 1028B to converge and form holographic surface 1018B. Therelayed holographic surface 1018B is slightly larger than the sourceholographic surface 1016A, due to magnification performed by the concavemirror corresponding to the optical path length between holographicsurface 1016A and the mirror. In an embodiment, the mirror is aspherical mirror, and the path length between the holographic surface1016A and the mirror 1007A is slightly smaller than the radius ofcurvature of the surface of mirror 1007A. In addition, the depthordering of the holographic surfaces is conserved by the relay: thesource surface 1016A is projected to be in front of the screen plane1021, and its relayed surface 1018B is also projected in front ofvirtual screen plane 1022. The source surface 1015A is projected behindthe screen plane 1021, and its relayed surface 1017B is also projectedbehind the virtual screen plane 1022, further from the viewer in eachcase. Thus, the depth reversal that occurs with the retroreflector inFIG. 5B has been avoided by using the mirror 1007A. Finally, because animage generated by the concave mirror 1007A is flipped, the relayedholographic sphere 1018B is projected to a position beneath the relayedholographic box 1017B, in opposite order to the position of thesesurfaces that appears in FIG. 5B. The set of relayed light paths 1028B,1032B may be considered as having been determined according to a second4D function defined by the relay system 5040, such that each relayedlight path has a set of positional coordinates (X,Y) and angularcoordinates (U,V) in a second 4D coordinate system defined with respectto a virtual screen plane 1022. The magnification, minification, andposition changes of the relayed surfaces 1018B and 1017B are all theeffect of the application of the second 4D function in the second 4Dcoordinate system.

In order to generate the relayed holographic surfaces shown in FIG. 5Bto a viewer 1050, some corrections may be made to the holographicsurfaces projected by the display shown in FIG. 5C. In an embodiment,the light field display 1001 may include a controller 190 configured toreceive instructions for accounting for the second 4D function byoperating the light field display 1001 to output projected lightaccording to the first 4D function such that the positional coordinatesand angular coordinates in the second 4D coordinate system for each ofthe set of relayed light paths 1028B and 1032B allow the relayedholographic surfaces 1018B and 1017B, respectively, to be presented to aviewer as intended. FIG. 5D illustrates an embodiment of some changesthat may be made to the projected objects in the display system of FIG.5C to correct for the optical effect of the relay system 5040. FIG. 5Dshows the position and magnification of the holographic surfaces thatwould have to be generated by the light field display 1001 if a relaysystem 5040 with a curved mirror configuration shown in FIG. 5D is usedin order to display much the same holographic objects that a viewer 1050would see in FIG. 5B. Holographic surface 1015A in FIG. 5C would have tobe projected to the position of holographic surface 1015C in FIG. 5D andmade slightly smaller to compensate for the magnification that resultsfrom the surface 1015C being a closer distance to the mirror 1007A.Holographic surface 1016A in FIG. 5C would have to be projected into theposition of holographic surface 1016C in FIG. 5D and magnified tocompensate for the minification of the relayed holographic surface thatoccurs at a greater distance from the mirror 1007A. The positions ofholographic surfaces 1015C and 1016C are right-left swapped, relative to1015A and 1016A in FIG. 5C to account for the inversion of the imagethat occurs with reflection due to the mirror. The result is thatholographic surface 1015C is relayed into 1017C, in precisely the sameplace as 1017A in FIG. 5B, and holographic surface 1016C is relayed into1018C, in precisely the same place as 1018A in FIG. 5B.

In FIG. 5D, the group of light rays along projected light paths 1036C,which form the projected holographic sphere surface 1016C, are comprisedof light rays 1041C, 1042C, and 1043C. These light rays are reflected bythe image combiner 101 into light paths 1037C, which are reflected bythe mirror 1007A into light ray group 1028C, comprised of light rays1041D, 1042D, and 1043D, and forming the relayed holographic surface1018C. In a similar way, in FIG. 5B, the group of light rays alongprojected light paths 1036A from the holographic sphere surface 1016Amap to the group of light rays along relayed light paths 1028A that formthe relayed holographic surface 1018A. Upon close inspection of FIG. 5B,the middle ray 1042A projected normal to the screen plane 1021 (ordisplay surface 1021) in FIG. 5B, often associated with a light fieldangular coordinate (u, v)=(0, 0), maps to the middle ray 1042B which isnormal to the virtual screen plane 1022 viewed by viewer 1050. In otherwords, for the retroreflector configuration shown in FIG. 5B, the lightray produced at (u, v)=(0, 0) is preserved, despite the fact that theangular coordinates u and v may be swapped as shown in FIG. 2B tocorrect the reversal of depth. However, in the curved mirror relayconfiguration shown in FIG. 5D, where no reversal of depth occurs, thecenter light ray 1042C in the group of projected light rays 1036Cprojected normal to the screen plane 1021 of light field display 1001,often associated with a light field angular coordinate (u, v)=(0, 0),maps to the middle ray 1042D which may not be normal to the virtualscreen plane 1022 viewed by viewer 1050. This is the same behavior thatis shown in FIG. 4D, where light rays 490C and 491C projected normal tothe display surface 497 produce light rays 490E and 491E, respectively,which generate angles θ₁ and θ₂ that vary with respect to the normal tothe virtual screen plane 469, depending in part on the location the raysintersect the holographic surface 488. The result is that if this isuncorrected, the viewer will not see the correct light field informationfrom the light ray 1042D. In the example that a specular highlight isprojected by the light field display 1001 in FIG. 5D along light rayalong the projected light path 1042C, this specular highlight willappear on relayed light path 1042D at an angle to the normal of virtualscreen plane 1022. To correct for this, the color and intensityinformation that is projected on the (u, v)=(0, 0) ray along projectedpath 1042C in absence of relay system 5040 should instead be projectedon light ray along the projected path 1043C if the relay system 5040 isin place so that this information will appear on mapped ray along thecorresponding relayed path 1043D, which is the (u, v)=(0, 0) rayrelative to the virtual screen plane 1022 and the observer 1050. Inother words, some remapping of light field coordinates may be made onthe light field display 1001 (in addition to the magnificationadjustments previously described) in order to relay a holographicsurface using a relay optical configuration with a curved mirror 1007A.Similarly, in FIG. 5D, light rays 1030C projected by the light fielddisplay 1001 and forming holographic object 1015C may also have a centerray at (u, v)=(0, 0). These light rays 1030C are directed into lightrays 1031C by the image combiner 101, which are then reflected intolight rays 1032C which pass through the image combiner 101 and convergeto help form relayed holographic object 1017C, with the center ray nolonger perpendicular to the virtual screen plane 1022. In FIG. 5D, thelight paths 1030C forming projected holographic object surface 1015C andlight paths 1036C forming projected holographic surface 1016C are eachdetermined according to a four-dimensional function defined by the lightfield display 1001 such that each projected light path has a set ofspatial coordinates and angular coordinates in a first four-dimensionalcoordinate system with respect to the light field display screen plane1021. The holographic surfaces 1015C and 1016C are relayed to relayedsurfaces 1017C and 1018C, respectively, wherein relayed locations of therelayed image surfaces 1017C and 1018C are determined according to asecond 4D function defined by the relay system 5040, such that lightpaths from the light field display 1030C, 1036C are relayed alongrelayed light paths 1032C, 1028C, each having a set of spatialcoordinates and angular coordinates in a second 4D coordinate system,respectively. In an embodiment, the light field display 1001 comprises acontroller 190 configured to receive instructions for accounting for thesecond 4D function by operating the light field display 1001 to outputlight according to the first 4D function such that the positionalcoordinates and angular coordinates in the second 4D coordinate systemfor the relayed light paths 1032C, 1028C allow the relayed imagesurfaces 1017C and 1018C to be presented to a viewer 1050 as intended.

Under the circumstance where the LF display 1001 produces unpolarizedlight, and an unpolarized 50% beam splitter 101 is used, about half thelight from holographic surfaces 1015C and 1016C is lost upon the firstpass through the beam splitter 101, and another half of the light islost upon the second pass through the beam splitter 101, resulting in nomore than 25% of the light from the holographic surfaces 1015C and 1016Cbeing relayed. If a polarized beam splitter 101 is used, then it ispossible that half of unpolarized light from the holographic surfaces1015C and 1016C is lost upon the first reflection from the beam splitter101, but the remaining light directed toward the mirror 1007A will be ina known first state of linear polarization. With a quarter wave retarderused for the optional optical element 1041A, the light returning fromthe mirror may be mostly in a known second state of linear polarization,orthogonal to the first state, and mostly be transmitted through thepolarized beam splitter 101, contributing to the relayed holographicsurfaces 1017C and 1018C. Under these circumstances, between 25% and 50%of the light from the holographic surfaces 1015C and 1016C may berelayed to holographic surfaces 1017C and 1018C. If the light fielddisplay 1001 produces polarized light, this efficiency can be increasedsubstantially with the use of a polarized beam splitter 101 and aquarter wave retarder 1041A.

The relay 5040 of the configuration shown in FIG. 5D may be used as oneor more of the relays in a holographic relay system comprised of tworelays, as shown in FIG. 3B. In FIG. 3B, both of the relays 130 and 140may be replaced with relay systems 5040, but in FIG. 3C, only relay 130may be replaced by relay 5040, since relay 140 requires light to betransmitted in two different directions. In another embodiment, twosubstantially identical relays 5040 are used in the holographic relaysystem configuration shown in FIG. 3B, and the effects of theminification, magnification, and rearranging of light field angularcoordinates (u, v) for the first relay 130 described above in referenceto FIG. 5D are at least partially reversed by the second relay 140.

In FIG. 5D, half of the light from light paths 1036C or 1030C from theholographic surfaces 1016C or 1015C, respectively, may be wasted sinceit passes through the beam splitter 101 into light rays alongtransmitted paths 1034 as shown in FIG. 5C. It is possible to addanother mirror 1007B, identical to mirror 1007A, placed opposite to thedisplay 1001A on the other side of the beam splitter 101, and orthogonalto mirror 1007A. FIG. 5E is an orthogonal view of a light field displaysystem comprising a holographic relay system 5050 comprised of a beamsplitter 101 and two concave mirrors 1007A, 1007B placed orthogonally toone another to achieve a high efficiency for light transmission fromprojected holographic surfaces to relayed holographic surfaces. Thisconfiguration is similar in concept to the second retroreflector 1006Bwhich appears in FIG. 1B. Although curved mirror 1007A is marked asoptional in the relay 5050 shown in FIG. 5E, the relay 5050 operateswith curved mirror 1007A present and curved mirror 1007B absent, curvedmirror 1007A absent and curved mirror 1007B present, or with both curvedmirrors 1007A and 1007B present. These variations of configurations ofrelay 5050 will be presented in this disclosure. With both curvedmirrors present, light rays along the projected paths 1036C fromholographic surface 1016C either are reflected by the beam splitter intoreflected light paths 1037C directed toward the mirror 1007A, or passthrough the beam splitter into transmitted light paths 1042A directedtoward the mirror 1007B. Light paths 1037C directed toward mirror 1007Areflect into light paths which are again incident on the beam splitter101, and a fraction of this light is transmitted through to relayedpaths 1028C (while the remaining fraction of this light incident on thebeam splitter 101, not shown, is directed downward back toward the lightfield display 1001). Light paths 1042A directed toward mirror 1007Breflect into light paths 1042B, which are incident on the beam splitter101, and a fraction of this light is reflected into paths 1028C,combining with the paths of light reflected by mirror 1007A (while theremaining fraction of this light, not shown, is transmitted through thebeam splitter 101 and directed back toward the light field display1001). The same is true for light from holographic surface 1015C, beingrelayed into holographic surface 1017C, but these light paths are notshown in FIG. 5D. In an embodiment, the concave mirrors 1007A and 1007Band the beam splitter 101 are aligned such that the light along paths1028C reflected from mirrors 1007A and 1007B substantially overlap.

Under the circumstance where the LF display 1001 produces unpolarizedlight, and an unpolarized 50% beam splitter 101 is used, almost all thelight from holographic surfaces 1015C and 1016C is directed to eithermirror 1007A or 1007B. Upon returning, at most half of the lightreflected from each mirror may be transmitted through the beam splitter101 toward the display, and not contribute to imaging of relayedholographic surfaces 1016C or 1017C. This gives an upper limit of 50% ofefficiency for light from holographic surfaces 1015C and 1016C to berelayed to holographic surfaces 1017C and 1018C. However, using apolarization beam splitter as well as a quarter wave retarder as theoptional optical elements 1041A and 1041B, as described in thediscussion of FIG. 1A as well as FIG. 5D, a substantially higherefficiency may result, since most of the light directed toward eachmirror has a specific linear polarization which may be rotated by 90degrees on its return trip back toward the beam splitter, resulting inmost of the light of two different reflected polarizations beingrecombined as it is directed to the relayed holographic surfaces 1017Cand 1018C.

In some embodiments, the focusing function of the mirrors 1007A and1007B shown in FIGS. 5C-5E may be replaced with one or more opticalelements such as lenses, mirrors, or some combination of these elements.In one embodiment, the entire relay system 5040 of FIGS. 5C-5D may bereplaced with a relay formed with one or more lenses such as the lensrelay system 5070 shown in FIG. 4E.

It is possible to use more compact Fresnel mirrors in place of thecurved mirrors 1007A and 1007B in FIG. 5E. FIG. 5F is an orthogonal viewof a light field display with a holographic relay system 5060 comprisedof a beam splitter 101 and two reflective Fresnel mirrors 1008A, 1008Bplaced orthogonally to one another to achieve a high efficiency forlight transmission from projected holographic surfaces to relayedholographic surfaces. This relay 5060 configuration is the same as therelay 5050 configuration of FIG. 5E, except the curved mirrors 1007A and1007B have been replaced with Fresnel mirrors 1008A and 1008B. Thenumbering of FIG. 5E applies to FIG. 5F, and the operation of relay 5060with Fresnel mirrors is very similar to the operation of relay 5050 withcurved mirrors. Although Fresnel mirror 1008A is marked as optional inthe relay 5060 shown in FIG. 5F, the relay 5060 operates with Fresnelmirror 1008A present and Fresnel mirror 1008B absent, Fresnel mirror1008A absent and Fresnel mirror 1008B present, or with both Fresnelmirrors 1008A and 1008B present. These variations of the configurationof relay 5060 will be presented in this disclosure.

Many of the display systems in this disclosure are designed to relaylight from one or more light sources through a relay system and to anobserver. For the purposes of avoiding unwanted scattering andreflection within these display systems, it is best to avoid directinglight into the display system in a direction opposite to the directionof the light from relayed objects observed by one or more viewers. It isnot always possible to keep the viewing area for relayed objectspresented by a display system in the dark. FIG. 5G shows the displaysystem of FIG. 5F confined to a light blocking enclosure 1080 with apolarization filter 1081 used as a window in the path of relayed lightpaths 1037E forming the surface 1018C of a relayed holographic object.The numbering of FIG. 5F is used in FIG. 5G. The polarization filter1081 may only pass light 1037E of a first state of polarization (denotedby the solid lines 1037) while absorbing the remainder of the light (notshown). The environmental light source 1085 produces light of twopolarizations 1091 (denoted by dot-dashed lines), but a light sourcepolarization filter 1082 only allows light 1092 of a second state ofpolarization (denoted by dashed lines) to pass through and illuminatethe environment around the display system 5055, and this light will notpass through the polarization filter 1081 window of the display system5055. This means that the environmental ambient light 1092 cannot enterinto the display system 5055 and reflect or scatter from elements withinthe relay or any other components in display system 5055. In anembodiment, a polarized light source 1085 may be used without a lightsource polarization filter 1082. It should be appreciated that theambient light rejection system formed by ambient light polarizationfilter 1082, the light blocking enclosure 1080, and the display systempolarization filter window may be used for any of the display systemswith relays presented in this disclosure.

Within display system 5055 in FIG. 5G, the light rays 1036C formingprojected holographic object 1016C may be of unpolarized light, denotedby dot-dashed lines. These light rays 1036C pass through an optionaloptical element 1083 and are partially reflected into light rays 1037Cby the image combiner 101 and partially transmitted 1036D through theimage combiner. The deflected light rays 1037C pass through the optionaloptical element 1041A and reflect from Fresnel mirror 1008A into lightrays 1037D. The portion of the light rays 1037D in a first state ofpolarization are passed by the polarization filter window 1081, whilethe portion of the light rays 1037D that are in an orthogonal secondstate of polarization are absorbed by the polarization filter window1081. Environmental light 1092 of a second state of polarization cannotenter through the polarization filter window 1081, eliminating thechance for reflection of these unwanted rays of light within the displaysystem 5055 and back out of the display system to the observer 1050. Theoptional optical elements 1083 and 1041A within the display system 5055may be used to control polarization in a more purposeful manner. Forexample, it may be desirable to minimize the fraction of light 1036Cwhich is passed directly through the image combiner 101 into light rayssuch as 1036D, since light rays such as 1036D can reflect from surfaceswithin the enclosure 1080 and exit the enclosure 1080 through thepolarization filter window 1081 as scattered light.

FIG. 5H shows the display system of FIG. 5G with a display polarizationfilter 1083 used in the path of the light field display, a quarter waveretarder used in the path of light rays which approach and reflect fromthe Fresnel mirror 1008A, and a polarization beam splitter 101. Thelight field display may project unpolarized light, and the displaypolarization filter 1083 may only pass light of a second state ofpolarization, denoted by the dashed lines 1036C. In an embodiment, thelight field display 1001A may produce only light of a secondpolarization, and the polarization filter 1083 is not needed. Apolarization beam splitter may be use as image combiner 101, wherein thepolarization beam splitter passes a first state of polarization anddeflects a second state of polarization. Since the incident light 1036Cis only of a second state of polarization, almost all the light 1036C isdeflected toward the Fresnel mirror 1008. The light of a second state ofpolarization 1037C (dashed lines) is mostly converted into reflectedlight 1037D of a first state of polarization (solid lines) by passingthrough the quarter wave retarder 1041A, reflecting from the surface ofa mirror 1008A, and passing through the quarter wave retarder 1041A onceagain. The light 1037D passes through the polarization filter window1081 into light rays 1037E of a first state of polarization (solidlines) to form relayed holographic object surface 1018C. Ambient light1092 of a second state of polarization (dashed lines) cannot enter intothe display system 5055 through polarization filter window 1081,avoiding unwanted scatter.

FIG. 6 shows an embodiment of a display system which relays holographicsurfaces projected by a light field display 1001 using a transmissivereflector 5030 as shown in FIG. 3A. The light field display 1001projects out-of-screen holographic surface 1016A on the viewer side 1010of the screen plane 1021, and in-screen holographic surface 1015A on thedisplay side 1011 of the screen plane 1021. Projected light rays alongthe projected light paths 1036A that converge on the surface ofholographic surface 1016A, and projected light rays along the projectedlight paths 1030A that converge at in-screen holographic surface 1015A(see the ray trace lines 1033) all diverge as they approach thetransmissive reflector 5030. The transmissive reflector 5030 ispositioned to receive light along the set of projected light paths1030A, 1036A and direct the received light along the set of relayedlight paths 1032A, 1028A respectively. In an embodiment, each of the setof projected light paths 1030A, 1036A has a set of positionalcoordinates (X,Y) and angular coordinates (U,V) in a four-dimensional(4D) coordinate system defined with respect to the display screen plane1021. In an embodiment, each light path in the set of relayed lightpaths 1032A, 1028A has a unique set of positional coordinates (X,Y) andangular coordinates (U,V) in a four-dimensional (4D) coordinate systemdefined with respect to the virtual screen plane 1022. Further, in anembodiment, an external surface 430 of the transmissive reflector 5030reflects a second portion of the received light along a set of reflectedlight paths 1130, 1136 in a second direction opposite the firstdirection. In an embodiment, a first portion of the light 1030A fromprojected holographic surface 1015A is received and relayed by relay5030 into light ray group 1032A which forms relayed holographic surface1017A, while a second portion of the light 1030A is reflected from thesurface 430 of relay 5030 into light rays 1130, where the relayed lightrays 1032A and the corresponding reflected light rays 1130 substantiallyoverlap, allowing both viewers 1050 and 1350 to observe the sameholographic surface 1017A. Similarly, a first portion of the light 1036Afrom projected holographic surface 1016A is received and relayed byrelay 5030 into light ray group 1028A which forms relayed holographicsurface 1018A, while a second portion of the light 1036A is reflectedfrom the surface 430 of relay 5030 into light rays 1136, where therelayed light rays 1028A and the corresponding reflected light rays 1136substantially overlap, allowing both viewers 1050 and 1350 to observethe same holographic surface 1018A. Observers 1050 and 1350 will observethe holographic surface as it were really there—so if the surface of aperson's face 1016A is being projected such that the correspondingrelayed holographic surface 1018A appears to be a depth-reversed face toviewer 1050, the face will appear to have normal depth to the opposingviewer 1350.

Notice that projected surface 1015A is further from the viewer thanprojected surface 1016A, but is relayed into relayed surface 1017A whichis closer to the viewer than the other relayed object 1018A. Thevertical distance between holographic surface 1016A and the relay 5030D1 is substantially the same as the horizontal distance between itscorresponding relayed holographic surface 1018A and the relay 5030.Similarly, the vertical distance D2 between holographic surface 1015Aand the relay 5030 is substantially the same as the horizontal distancebetween its corresponding relayed surface 1017A and the relay 5030. Anobserver 1050 will see holographic surface 1017A floating in space nextto but closer than holographic surface 1018A. An observer 1350 will seethe holographic surface 1018A floating in space next to but closer toholographic surface 1017A. If the holographic source surfaces 1015A and1016A are rendered prior to being displayed in order to achieve thecorrect depth ordering of relayed holographic surfaces 1017A and 1018Aas observed by viewer 1050, which means the depth of surfaces isreversed about the screen plane 1021 and the light field angularcoordinates U-V are reversed as shown in FIGS. 2B and 2C, and discussedin reference to FIGS. 1A and 5B above, then the U-V coordinates will bereversed for the surfaces reflected from the surface of transmissivereflector 5030 and observed at 1350. In other words, the depth may notappear correctly for holographic surface 1017A or 1018A for an observer1350 viewing light rays 1130 or 1136, respectively. To correct for this,it is possible to place a correction optical element similar to thatshown in FIG. 2A at the plane 1137 in order to perform U-V coordinatereversal for the set of the reflected light paths 1130, 1136. In anotherembodiment, with a different light field rendering of holographicsurfaces 1015A or 1016A, and with no correction optical element at plane1137, the observer 1350 may perceive the holographic surfaces 1017A and1018A with the correct depth ordering, and a corrective optical element20 similar to that shown in FIG. 2A may be placed at the virtual displayplane 1022 to allow observer 1050 to also view the holographic surfaces1017A and 1018A with the correct depth ordering. In other words, if thecorrection optical element 20 like that shown in FIG. 2A is used toallow both observers 1050 and 1350 to see the holographic surfaces 1017Aand 1018A with the correct depth, they can be placed at plane 1022 or1137, depending on whether the light field rendering of holographicsurfaces from the light field display 1001 contains steps which reversethe depth around the screen plane 1021 by reversing the polarity of theU-V coordinates as shown in FIG. 2B.

FIG. 7 illustrates a holographic display system that is the same as theholographic system of FIG. 5B with the addition of another display 1201opposite the first display 1001. The numerical labeling from FIG. 5Bapplies to FIG. 7 . The relay system 5010 is comprised of an imagecombiner 101 and a retroreflector 1006A. If 1201 is a light fielddisplay, then the light field display 1201 may be configured as thelight field display 1001 discussed above with respect to FIGS. 1A, withone or more display devices 1202 containing a plurality of light sourcelocations, an imaging relay 1203 which may or may not be present whichacts to relay images from the display devices to an energy surface 1205,and an array of waveguides 1204 which project each light source locationon the energy surface into a particular direction in three dimensionalspace. The energy surface 1205 may be a seamless energy surface that hasa combined resolution that is greater than any individual display device1202, while plane 1221 is the screen plane of 1201, which may coincidewith the display surface. If 1201 is a traditional 2D display, thenrelays 1203 and/or waveguides 1204 may be absent. Display 1201 maydisplay a 2D image (not shown) or a holographic surface 1213. The raysalong an additional set of projected light paths 1231 leaving thedisplay 1201 reflect from the surface of the beam splitter 101, formingdiverging ray group along an additional set of relayed light paths 1233,which can be ray traced back through imaginary paths 1234 to reveal aconvergence point at a perceived holographic surface 1214. The verticaldistance D3 between the projected holographic surface 1213 and the beamsplitter 101 is substantially equal to the horizontal distance betweenthe beam splitter and the perceived holographic surface 1214. Anobserver 1050 will see holographic surfaces 1017A, 1018A, and displayedsurface 1214, which may or may not be holographic depending on whetherdisplay 1201 is a light field display. Using a 2D display as 1201, it ispossible to create a uniform background imaging plane that can be placedat any reasonable distance from the observer 1050 depending on thedistance between display 1201 and beam splitter 101. An occlusion system1207 with individually addressable occlusion elements may block somelight from the display 1201. The occlusion system 1207 may be comprisedof one or more of: a transparent LED panel, a transparent OLED panel, anLC panel, a portion of a LCD panel (e.g. without a backlight orreflectors), a parallax barrier, a real-world physical object, a maskplaced on a glass plane, or some other type of panel that may fully orpartially block light at select locations and or select angles. Theocclusion system 1207 can be placed in the path of display 1201 atdistance 1210 from the screen plane 1221 of display 1201 in order toblock some or all of the light from display 1201. The occlusion system1207 may be considered an occlusion barrier with individuallyaddressable occlusion regions which block all or a portion of the light1231 from display 1201. The occlusion system 1207 may be placed at thesame distance from the display as the projected holographic object 1213and have a position which is adjustable. The occlusion system 1207 canbe used to block out portions of the surface 1213 from reaching therelay 5010, in the event that relayed holographic surface 1017A orrelayed holographic surface 1018A occludes perceived holographic surface1214, and both images are not desired to be displayed at the same time.If the occlusion system 1207 is a portion of an LCD panel containing oneor more polarizers and a liquid crystal (LC) layer, the beam splittercan be a polarization beam splitter that is selected to reflect 100% ofthe polarized light passing through 1207. Similarly, an occlusion system1208 can be placed above light field display 1001 at a distance 1211 inorder to block all or some of the light from display 1001. The occlusionsystems 1207 and 1208 may not be necessary to avoid occlusion problemsif 1201 is a light field display, since coordinated rendering of both ofthe light field displays 1001 and 1201 can be used to avoid occlusion.In an embodiment, the display system shown in FIG. 7 may include acontroller 190 configured to issue display instructions to the lightfield display 1001 to output light according to a 4D function. Thecontroller 190 may issue coordinated instructions to the other display1201 and the occlusion system 1207 to present the holographic surfaces1017A, 1018A, and surface 1214 as intended. It is to be appreciated thevarious embodiments in above discussions with respect to FIG. 7 may beimplemented in part or in whole in other embodiments of the holographicdisplay systems of the present disclosure, including those in FIGS.4C-4D and FIGS. 5C-5D. For example, the second display 1201 andocclusion systems 1207 and 1208 discussed above may be implemented towork with a relay system that includes at least one concave mirror asdescribed in FIG. 5C.

FIG. 8A is a holographic display system that is the same as theholographic display system of FIG. 7 with the relay system 5010 replacedby transmissive reflector relay 5030. The numbering of FIG. 7 is used inFIG. 8A. A first portion of the projected light rays 1231 formingholographic object 1213 may partially reflect from the surface of thetransmissive reflector 5030, forming diverging ray group 1332. A secondportion of the projected light rays 1231 will be received and relayed tolight rays 1333 forming relayed holographic object 1314, where therelayed light paths 1333 substantially overlap with the reflected lightpaths 1332. The vertical distance D3 between the displayed surfaces 1213and the transmissive reflector relay 5030 may be substantially equal tothe horizontal distance between relay 5030 and the relayed holographicsurface 1314. An observer 1050 will see holographic surfaces 1017A,1018A, and displayed holographic surface 1314. In another embodiment,1201 is a 2D display rather than a light field display, and observer1050 sees holographic surfaces 1017A, 1018 in front of a 2D backgroundpositioned at virtual plane 1137. Using a 2D display as display 1201, itis possible to create a uniform background imaging plane that can beplaced at any reasonable distance from the observer 1050 depending onthe distance between display 1201 and transmissive reflector 5030. Theocclusion systems 1207 and 1208 may not be necessary to avoid occlusionproblems if 1201 is a light field display, since a controller 190 mayissue coordinated display instructions for both of the light fielddisplays 1001 and 1201 to support proper computational occlusion ofrelayed background objects 1018A, 1214 behind foreground objects 1017A.A corrective optical element 20 from FIG. 2A or similar configurationsthat reverse the polarity of the angular 4D light field coordinates U, Vmay be placed at virtual plane 1137 and not virtual plane 1337, orvirtual plane 1337 and not virtual plane 1137, or at both locations, orat none. Also, corrective optical element 20 placed at planes 1337 and1137 may both be moved closer or further away from the transmissivereflector 5030. Another option is to have corrective optics 20 from FIG.2A or similar configurations, which reverse the polarity of U, Vcoordinates placed just above the screen plane 1021 of the light fielddisplay 1001. Finally, system 130 can be built using a mirror in placeof transmissive reflector 5030, which may result in two independentviews at observer 1050 on the left of 5030 and an observer located onthe right of 5030 (not shown), where each observer would only be able tosee holographic surfaces from a single display. It is to be appreciatedthe various embodiments in above discussions with respect to FIG. 8 amay be implemented in part or in whole in other embodiments of theholographic display systems of the present disclosure, including thosein FIGS. 4C-4D and FIGS. 5C-5D. For example, the second display 1201 andocclusion systems 1207 and 1208 discussed above may be implemented towork with a relay system that includes at least one concave mirror asdescribed in FIG. 5C. In an embodiment, the display system shown in FIG.8A may include a controller 190 configured to issue display instructionsto the light field display 1001 to output light according to a 4Dfunction. The controller 190 may issue coordinated instructions to theother display 1201 and the occlusion system 1207 to present theholographic surfaces 1017A, 1018A, and surface 1314 as intended.

FIG. 8B shows an embodiment of the display system in FIG. 8A to performocclusion handling using the occlusion system 1207. The labels of FIG.8A apply to FIG. 8B. A portion 1367 of occlusion system 1207 may beactivated to block light 1361 from one side of projected holographicsurface 1213. Only the orthogonal rays 1362 from the surface 1213 areshown, and they partially reflect from the transmissive reflector 5030into rays 1364 that reach the observer 1050. The rays 1362 are relayedby 5030 into rays 1363, which form the projected holographic surface1366. Substantially no blocked light rays 1361 from the portion of thesurface 1213 are visible to observer 1050, corresponding to the blockedportion 1365 of the relayed holographic image 1366.

FIG. 8C shows an embodiment of a display system similar to that shown inFIG. 8A, with substantially all the rays of light that would reach anobserver 1350 on the right of transmissive reflector 5030, but omittingsome of the light rays that would reach an observer on the left of 5030(not shown) for clarity. The numbering of FIG. 8A applies to thisdrawing. Light rays 1030A forming holographic object 1015A reflect fromthe surface 430 of relay 5030 into light rays 1331, which are perceivedby observer 1350 to originate from the position of relayed holographicobject 1017A. Similarly, light rays 1036A forming holographic object1016A reflect from the surface 430 of relay 5030 into light rays 1337,which are perceived by observer 1350 to originate from the position ofrelayed holographic object 1018A. If the display 1201 is a holographicdisplay, then holographic surface 1213 will be relayed to holographicsurface 1314, and the observer 1350 will see 1314 in the foreground, andholographic surfaces 1017A and 1018A in the background. If the display1201 is a 2D display, then observer 1350 will see a flat foregroundimage, and holographic surfaces 1017A and 1018A in the background. Asdiscussed for FIG. 8A, if 1201 is a light field display, occlusionhandling may be done by coordinating the two light fields 1001 and 1201,or by using the occlusion systems 1207 and/or 1208. If 1201 is a 2Ddisplay, then occlusion handling may be done using the occlusion systems1207 and/or 1208.

Combining Images of Real-World Objects with Holographic Objects

With reference to at least FIGS. 3B, 3C, 8A, 8B, and 8C, the presentdisclosure contemplates and describes various embodiments for using arelay system to relay first and second image surfaces from first andsecond image sources, respectively. In an embodiment, the first imagesource may include the surface of a light field display, and the lightfrom the light field display may form the first image surface of aholographic object. In an embodiment, the second image source mayinclude a 2D display surface, a stereoscopic display surface, anautostereoscopic display surface, a multi-view display surface which maybe a horizontal parallax only multi-view display surface, the surface orsurfaces of a volumetric 3D display, a second light field displaysurface, the surface of a real-world object emitting light, or thesurface of a real-world object reflecting light. Correspondingly, theimage surface of the second image source may include an image surfaceprojected from a 2D display surface, an image surface projected from astereoscopic display surface, an image surface projected from anautostereoscopic display surface, an image surface projected from amulti-view display surface, an image surface of a volumetric 3D display,a surface of a holographic object formed by light paths projected from asecond light field display, a surface of a real-world object, or arelayed image of the surface of the real-world object.

In one embodiment, the relay system of the present disclosure may relaythe first and second image surfaces to relayed locations a distance awayfrom the first and second image surfaces, where first and second relayedimages surfaces are observable at the respective relayed locations. Forexample, in an embodiment, the relayed holographic objects and therelayed image of a real-world object may appear together (e.g. 121C,122C, and 123C shown in FIG. 3C). If a relayed holographic objectappears in front of a relayed image of a real-world object, then anocclusion system may be disposed proximate to the real-world object toblock off a portion of the light from the relayed image of thereal-world object that is being occluded by the holographic object sothat a viewer cannot seethe real-world object behind the holographicobject. This allows a presentation of the holographic object in front ofthe real-world image with current occlusion handling. This may helpavoid having an opaque relayed holographic object (e.g. a human headthat is not a ghost) appear transparent with the light from the relayedimage of a real-world object visible directly behind the relayedholographic object to an observer. In this disclosure, sometimes nodistinction is made between a relayed object and a relayed surface. InFIG. 8C, for example, the projected holographic objects 1015A and 1016Aare surfaces which are relayed by relay 5030 to relayed holographicsurfaces 1017A and 1018A, respectively. The projected holographic objectsurfaces 1015A and 1016A may be referred to as ‘projected holographicobject surfaces’, ‘projected holographic objects’, or ‘holographicobjects’ equally in this disclosure. The relayed holographic objectsurfaces 1017A and 1018A may be referred to as ‘relayed holographicsurfaces or ‘relayed holographic objects’ equally in this disclosure.

In some embodiment of the present disclosure, some relay systems areconfigured to reverse a depth profile of the image surface being relayed(e.g. relay system 5010 shown in FIG. 1A), and some relay systems areconfigured not to do so (e.g. relay system 5040 shown in FIG. 5D). Ifthe relay system performs depth reversal, then the relayed image of animage surface, such as a holographic object surface, will have a depthprofile different from that of the original image surface. In oneembodiment, the relay image surface may have an intended depth profileby configuring the original image surface to have a pre-reversed depthprofile; for example, a real-world object may be configured to have areversed depth profile so that the relayed image surface of the realworld object has the intended depth profile. In another embodiment, arelay system may include two relay subsystems, which each relayreversing depth, with the second relay subsystem reversing the depthreversal performed by the first relay subsystem, resulting in a relayedimage surface with substantially the same depth profile as the originalimage surface. For example, an image surface of a real-world object maybe relayed twice through two relay subsystems that reverse depth,thereby resulting in a relayed image surface of the real-world objectthat substantially maintains the same depth profile as the originalimage surface of the real-world object. In some relay systemembodiments, there is no depth reversal and depth reversal does not needto be addressed (e.g. relay system 5040 shown in FIG. 5D).

To illustrate the principles discussed herein, FIG. 9A shows anembodiment of a display system 9001 comprised of a relay system 9001which is similar to the relay system shown in FIG. 3C, wherein the lightfrom two holographic object surfaces 121A and 122A projected around ascreen plane 1021A of a light field display 1001A is combined with thelight from a real-world object 123A via first and second inputinterfaces of an optical combining system 101, and these three objectsare relayed to another location near a virtual display plane 1022B. Thenumbering of FIG. 3C is used in FIG. 9A for similar elements. In theembodiment shown in FIG. 9A, the relay system 5080 is configured toreceive light from at least one of the first image sources 1001A andsecond image sources 123A through a first relay subsystem 5030A of therelay system 5080, the first relay subsystem 5030A operable to relay thereceived light to define a first relayed image surface 121B/122B(relayed holographic objects) or 123B (relayed real-world objectsurface) corresponding to the respective image surface, the firstrelayed image surface having a depth profile different from a depthprofile of the respective image surface 121A/122A or 123A defined bylight from the at least one of the first and second image sources. In afurther embodiment, at least one of the first and second image sourcescomprises a real-world object 123A, wherein the first relay subsystem5030A is operable to receive light from a surface of the real-worldobject 123A and wherein the first relayed image surface 123B comprises arelayed image of the surface of the real-world object having a depthprofile different from a depth profile of the surface of the real-worldobject 123A. In another embodiment, the relay system 5080 furthercomprises a second relay subsystem 5030B configured to direct light fromthe first relayed image surface 121B/122B (relayed holographic objects)into the viewing volume near observers 1050A-C, thereby defining asecond relayed image surface 121C/122C of relayed holographic objectscorresponding to the respective image surface, and to relay light fromthe other one 123A of the at least one of the first and second imagethat is not projected from a holographic display to relayed locations123C in the viewing volume, thereby defining a first relayed imagesurface corresponding to the respective image surface 123A, the secondrelayed image surface 121C/122C having a depth profile that issubstantially the same as the depth profile of the respective imagesurface 121A/122A defined by light from the at least one of the firstand second image sources 1001A. In an embodiment, an image source iscomprised of the real-world object 123A, and the relay system 9001includes an occlusion system 150, which in an illustrated embodiment,may include one or more occlusion layers 151, 152, and 153, wherein theocclusion layers may block out some of the light rays from thereal-world object 123A, preventing these light rays from reaching therelay locations of the relayed real-world object image surface 123C. Inthis case, the relay subsystem 5080 may include a first transmissivereflector relay subsystem 5030A and second transmissive reflector relaysubsystem 5030B, each of which reverses the depth, so that the secondtransmissive reflector 5030B reverses the depth-reversal of the firsttransmissive reflector relay subsystem 5030A, such that the overallrelay system 5080 preserves the depth profile of the real-world object123A as well as the holographic object surfaces 121A and 122A. Theocclusion layers 151, 152, and 153 may contain a plurality of parallaxelements, which, in an embodiment, may be individually-addressed lightblocking elements. In one embodiment, the occlusion layers 151, 152, and153 may each be a portion of an LCD panel containing one or morepolarizers and a liquid crystal (LC) layer with individually-addressablepixels, a transparent OLED display panel with individually-addressablepixels, or another panel that may selectively occlude light and betransparent, semi-transparent, or light blocking.

The relayed locations 160 are locations where the relayed holographicobject surfaces 121C and 122C are distributed about a relayed virtualdisplay screen 1022B, and relayed image surface 123C of the real-worldobject 123A. A relayed image of a real-world object will appear to be aslife-like as a holographic object, since the light rays that leave thesurface of the real-world object such as 123A are transported by therelay system 5080 in the same way that the light rays leaving thesurface of holographic object 121A are transported to form holographicobject 121C. Controller 190 may generate display instructions for thelight field display 1001A as well as send configuration instructions tothe occlusion planes 151, 152, and 153.

FIG. 9B shows a first embodiment of an occlusion system 150, comprisingone or more layers of occlusion planes 151, 152, and 153 located closeto the real world object 123A, and designed to block the portion of thelight from the real-world object 123A that would pass through aprojected holographic object 121AE and reach three observer positions1050AE, 1050BE, and 1050CE. Holographic object 121AE is shown torepresent the location of holographic object 121A relative to real-worldobject 123A once the light rays 131A from projected holographic object121A are combined with the light rays 133Y from real-world object 123Aby the optical combiner 101. In other words, projected holographicobject 121AE is shown in the equivalent optical location of holographicobject 121A relative to real-world object 123A. The three observerpositions 1050AE, 1050BE, and 1050CE correspond to the viewing positions1050A, 1050B, and 1050E of the relayed image surfaces shown in FIG. 9A,respectively, and appear in the opposite top-down order because therelayed real-world image surface 123C is up-down flipped relative to thereal-world object 123A. A pattern of individually-addressablelight-blocking elements 188 may be actuated on each occlusion plane 151,152, and 153 in order to block the portion of light rays from thereal-world object 123A passing through a holographic object 121AE andreaching three different viewing locations. This includes blocked lightrays 943A of the light rays 933A reaching observer 1050AE, blocked lightrays 943B of the light rays 933B reaching observer 1050BE, and lightrays 943C of the light rays 933C reaching observer 1050CE. The patternof light-blocking elements may be determined computationally oralgorithmically, and may be updated at the same video frame refresh rateof the holographic display 1001A in FIG. 9A in order for relayedholographic object surface 121C to be perceived by observers 1050A,1050B, and 1050C to continually occlude the relayed real-worldbackground image surface 123C, even as the relayed holographic objectsurface 121C is moved relative to the relayed background image surface123C of a real-world object in FIG. 9A. It is also possible that aportion of the relayed holographic object surface 121C may appear to besemi-transparent to the background image surface 123C of a relayedreal-world object, in which case the corresponding occlusion regions 188may be semi-transparent rather than opaque.

FIG. 9C shows a second embodiment of an occlusion system 150, comprisedof one or more layers of occlusion planes 151, 152, and 153 located ashort distance from the real-world object 123A, and designed to blockthe portion of the light from the real-world object 123A that would passthrough projected holographic object surface 121AE and reach threeobserver positions 1050AE, 1050BE, and 1050CE. The numbering for FIG. 9Bis used in FIG. 9C for similar elements. In the embodiment shown in FIG.9C, two of the occlusion planes 152 and 153 are located at substantiallythe same position corresponding with the holographic object 121AE, andthe selected occlusion regions 188 on each panel are activated so thatthey overlap with the holographic object 123AE. The occlusion regions188 may be determined computationally or algorithmically, and may beupdated at the same video frame rate of the holographic display 1001A inFIG. 9A in order for relayed holographic object surface 121C to beperceived by observers 1050A, 1050B, and 1050C to continually occludethe relayed real-world background image surface 123C, updated insynchronization to the movement of relayed holographic object surface121C relative to the relayed background image surface 123C of thereal-world object 123A. If a portion of the relayed holographic object121C should appear to be semi-transparent to the background relayedimage surface 123C of a real-world object, the corresponding occlusionregions 188 may be configured to be semi-transparent rather than opaque.To account for movement of the holographic surface 121A relative to thereal-world object 123A, one or more occlusion planes 151, 152, and 153may be mounted on a motorized translation stage so they can be placed atthe same effective position of holographic surface 121A as it moves.

FIG. 9D shows the effect of the occlusion layers of the occlusion system150 shown in FIG. 9C on the relayed real-world object image surface121C, as viewed by observer positions 1050A, 1050B, and 1050C shown inFIG. 9A. The dashed outlines 152E and 153E are relayed images of theocclusion layers 152 and 153 shown in FIGS. 9A and 9C, respectively. Therelayed regions 188B of occlusion on these relayed images of planes 152and 153 show where occlusion sites may be selected to provide theocclusion of relayed surface 123C by relayed holographic surface 121C.Observer 1050A cannot see the portion 161A of relayed image surface 123Cof the real-world object 123A that lies behind the relayed holographicobject surface 121C because relayed light rays from source 123A that liebetween light rays 943D are blocked by occlusion sites activated onocclusion planes 152 and 153 shown in FIG. 9A. Similarly, observer 1050Bcannot see portion 161B of relayed real-world image surface 123C behindrelayed holographic object surface 121C, as relayed light rays fromsource 123A between light rays 943E are blocked by occlusion sitesactivated on occlusion planes 152 and 153 shown in FIG. 9A. Observer1050C cannot see portion 161C of relayed real-world image surface 123Cbehind holographic object 121C, as relayed light rays from source 123Abetween light rays 943F are blocked by occlusion sites activated onocclusion planes 152 and 153 shown in FIG. 9A. In the example shown inFIGS. 9C and 9D, no occlusion handling is shown to be performed forholographic object 122C, although this is possible to happensimultaneously with the occlusion handling of holographic object 121C.The occlusion regions 188 on occlusion planes 151, 152, and 153 may beupdated continuously so that light from real-world object 123A iscontinuously occluded by relayed holographic objects such as 121C and122C in such a way that those holographic objects look like they arelife-like objects moving in front of an actual background formed withrelayed real-world object surface 123C, with occlusion handled properlyfor all viewers of the relayed object 121C, 122C, and 123C. It is alsopossible that the relayed holographic object surfaces such as 121C and122C appear to be semi-transparent to the relayed background imagesurface 123C of real-world object 123A, which in case the occlusionregions 188 may be semi-transparent, only attenuating rather thancompletely occluding portions of the light from real-world object 123A.And finally, the one or more occlusion planes 151, 152, and 153 may bemotorized so they can be moved to optically coincide with one or severalprojected holographic objects 121A and 121B even if they changeposition.

FIG. 9E is the display system of FIG. 9A with the occlusion system 150replaced by a real-world occlusion object 155A which blocks unwantedlight rays from the real-world object 123A. The numbering of FIG. 9A isused in FIG. 9E. The real-world occlusion object 155A may be similar inshape or profile to at least one projected holographic object 122A andmay be painted or coated with a light absorbing material such as matteblack paint. As shown in FIG. 9E, because the real-world occlusionobject 155A has been positioned so that it is equidistant from the imagecombiner 101 as the projected holographic object 121A, the surface ofreal-world occlusion object 155A will be relayed to relayed surface 155Cby the relay system 5080 so that it coincides at substantially the samelocation as the relayed surface 121C of the projected holographic objectsurface 121A. The light rays 157A and 158A from the real-world object123A are almost occluded by the edges of the occlusion object 155A andare relayed into light rays 157C and 158C by the relay system 5080,respectively. Relayed light ray 158C will be observed by observer 1050A,but light rays from relayed object 123C parallel to light ray 158C thatare just below light ray 158C will be blocked by real-world occlusionobject 155A before they are relayed by relay 5080. The result is thatthe portion of the relayed surface 123C will not be visible behindrelayed holographic surface 121C from the viewpoint of observer 1050C.Similarly, relayed light ray 157C will be seen by observer 1050A, butlight rays from relayed object 123C which are parallel to light ray 157Cand just above 157C will also be blocked by real-world occlusion object155A before they are relayed by relay 5080. The result is that theportion of the relayed surface 123C will not be visible behind relayedholographic surface 121C from the viewpoint of observer 1050A. Insummary, FIG. 9E shows that in a display system in which the light froma projected holographic surface 121A and a real-world object surface123A are combined and relayed, then a real-world occlusion object 155Awith the same dimensions as the dimensions of the relayed holographicobject surface 121B may be placed in a location which blocks a portionof the light from the real-world object 123A such that the relayedholographic object surface 121C and the relayed surface of real-worldocclusion object 155C are coincident, the real-world occlusion object155A offering occlusion of the relayed real-world object surface 123Cbehind the relayed holographic object surface for all viewers 1050A-Cwithin the FOV of the relayed objects 121C and 123C. In an embodiment,the real-world occlusion object 155A has its location controlled by amotorized positioning stage (not shown), and 155A can be moved 156 incoordination with the movement of a projected holographic object 121A sothat the relayed position 155C of relayed occlusion object 155Acontinually coincides with the position of a relayed holographic objectsurface 121C. A controller 190 may simultaneously issue displayinstructions to the light field display 1001A as well as issue commandsto a motion controller in order to direct coordinated movement 156 ofthe real-world occlusion object 155A as well as movement of a projectedholographic object 121A.

FIG. 9F shows the effect of the real-world occlusion object 155A shownin FIG. 9E on the relayed real-world object image surface 123C, asviewed by observer positions 1050A, 1050B, and 1050C shown in FIG. 9E.The relayed surface 155C of the real-world occlusion object 155A issubstantially coincident with the relayed surface 121C of projectedholographic object 121A. Observer 1050A cannot see the portion 162A ofrelayed real-world image surface 123C of the real-world object 123A thatlies behind the relayed holographic object surface 121C because relayedlight rays from source 123A that lie between light rays 943D are blockedby the occlusion object 155A. Similarly, observer 1050B cannot seeportion 162B of relayed real-world image surface 123C behind relayedholographic object surface 121C because relayed light rays from source123A that lie between light rays 943E are blocked by real-worldocclusion object 155A. Finally, observer 1050C cannot see portion 162Cof relayed real-world image surface 123C behind holographic object 121Cbecause relayed light rays from source 123A that lie between light rays943D are blocked by real-world occlusion object 155A shown in FIG. 9E.

FIG. 9G is a display system 9002 in which an observer sees the relayedsurface of a holographic object projected in front of the relayedsurface of a real-world background object or a background display, withno depth reversal of the relayed objects and proper occlusion handlingfor the background surface behind the relayed foreground holographicsurface. The relay system of FIG. 9G is similar to the relay system ofFIG. 9A, but while the real-world object or display is relayed throughtwo transmissive reflectors in both configurations, in FIG. 9G theholographic object 121G is inserted into the optical path along with thelight from the real-world background object or display 123F at alocation between the two transmissive reflectors. In FIG. 9G, thesurface of a real-world object or a display 123F is relayed to relayedobject surface 123H by the relay system 5090 comprised of two relaysubsystems with transmissive reflectors 5030F and 5030G as well as imagecombiner 101F.

The relay 5090 shown in FIG. 9G is comprised of two transmissivereflectors 5030F, 5030G placed on parallel planes and separated from oneanother with an image combiner 101F disposed between them. The firsttransmissive reflector relay subsystem 5030F offers a first inputinterface configured to receive light from a first image source which isthe surface of real-world object or 2D display 123F and is operable torelay the received light to a define a first relayed image surface 123Gand be received by an image combiner 101F, the first relayed imagesurface 123G having a depth profile different from a depth profile ofthe respective image surface 123F. The relay system 5090 furthercomprises an image combining element positioned to combine light fromthe first relay subsystem 5030F forming the relayed surface 123G ofreal-world object or display surface 123F, and light 131G from secondimage source 1001F defining a holographic surface 121G, wherein thecombined light comprising the first relayed image surface 123G and theholographic surface 121G is directed to the second relay subsystem 5030Gwhich is configured to relay the combined light to the viewing volume135 near viewer 1050G. The image combiner 101F offers a second interfaceto receive light from the second image source light field display 1001F,and this light is combined with the light from the second image sourceand relayed to a viewing volume 135 near viewer 1050 by the secondtransmissive reflector relay subsystem 5030G. The surface of real-worldobject or display 123F is relayed twice, first to 123G followed by asecond relay to 123H, while the surface of projected holographic object121G is relayed once to 121H. For this reason, the depth profile of theonce-relayed holographic surface 121G is reversed, while the depthprofile of the twice-relayed holographic surface 123H of real-worldobject or display 123F is not reversed. In an embodiment, holographicsurface 121G defined by light paths 131G projected from the light fielddisplay 1001F has a first projected depth profile with respect to screenplane 1021F, and the holographic surface 121G is relayed by the relaysystem to define first relayed image surface 121H comprising a relayedholographic surface with a first relayed depth profile that is differentfrom the corresponding first projected depth profile of 121G.

In an embodiment, the relay system 5090 is configured to receive lightfrom one of the first and second image sources 123F that is not aholographic display through a first relay subsystem 5030F of the relaysystem 5090, the first relay subsystem 5030F operable to relay thereceived light to define a first relayed image surface 123Gcorresponding to the respective image surface 123F, the first relayedimage surface 123G having a depth profile different from a depth profileof the respective image surface 123F defined by light from the one ofthe first and second image sources which is not a holographic object. Inanother embodiment least one of the first 123F and second 1001F imagesources comprises a real-world object 123F wherein the first relaysubsystem is operable to receive light from a surface of the real-worldobject 123F, and wherein the first relayed image surface 123G comprisesa relayed image surface of the real-world object having a depth profiledifferent from a depth profile of the surface of the real-world object123F. In another embodiment, the relay system 5090 further comprises asecond relay subsystem 5030G configured to direct light from the firstrelayed image surface 123G to the viewing volume 135 near observer1050G, and to relay light from the at least one of the first and secondimage sources defining a holographic surface 121G to relayed locationsin the viewing volume 135, thereby defining a relayed image surface 121Hof the holographic surface. In another embodiment, the relay systemfurther comprises an image combining element 101F positioned to combinelight 133E from the first relay subsystem and light from the at leastone of the first and second image sources defining a holographic surface121G, wherein the combined light 133E and 133H comprising the firstrelayed image surface 123G and the holographic surface 121G is directedto the second relay subsystem, which is configured to relay the combinedlight to the viewing volume 135. In an embodiment, the second relayedimage surface 123H comprises a second relayed image surface of thereal-world object 123F, the second relayed image surface 123H of thereal-world object having a depth profile that is substantially the sameas the depth profile of the surface of the real-world object 123F.

In an embodiment, the light field display comprises a controller 190configured to issue instructions for accounting for the differencebetween the first projected depth profile 121G and the first relayeddepth profile 121H by operating the light field display 1001A to outputprojected light such that the first relayed depth profile of the firstrelayed image surface is the depth profile intended for a viewer. Inanother embodiment, relayed locations of the first relayed image surface121H are determined according to a second 4D function defined by therelay subsystem 5030G, such that light from the light field display1001F is relayed along relayed light paths 131J each having a set ofspatial coordinates and angular coordinates in a second 4D coordinatesystem, wherein the light field display 1001F comprises a controller 190configured to receive instructions for accounting for the second 4Dfunction by operating the light field display 1001F to output lightaccording to the first 4D function such that the positional coordinatesand angular coordinates in the second 4D coordinate system for therelayed light paths 131J allow the first relayed image surface 121H tobe presented to a viewer as intended.

The optical system 9002 shown in FIG. 9G offers first and second inputinterfaces for first and second sets of light paths from first imagesource 123F and second image source 1001F respectively. The second setof light paths 131G are determined according to a four-dimensionalfunction defined by the light field display 1001F such that eachprojected light path has a set of spatial coordinates and angularcoordinates in a first four-dimensional coordinate system defined withrespect to a display screen plane 1021F of display 1001F, wherein thelight from the first image source 123F is operable to define a firstimage surface 123FS. The first input interface is relay subsystem 5030Fconfigured to receive light along a first set of light paths 133D from afirst image source 123F which in this example is a display or real-worldobject 123F, wherein the light from the first image source 133D isoperable to define a first image surface 123FS which is the surface ofreal-world object or display 123F. The second relay subsystem 5030G isconfigured to direct the received light from the first 123F and second1001F image sources to a viewing volume 135, wherein at least one and inthis case both of the first image surface 123FS and second image surface121G are relayed by the relay system into the viewing volume 135 asrelayed first surface 123H and relayed second holographic surface 121H,respectively. The side view detail 9003 of FIG. 9G taken from observerviewpoint 1050F shows that light from a second image source of a lightfield display 1001F forms projected holographic surface 121G, where itis combined with the relayed light 133E from the real-world object ordisplay 123F in between the two transmissive reflectors 5030F and 5030G,and relayed to relayed holographic surface 121H by relay subsystem5030G. The observer 1050G will see the relayed holographic surface 121Hin front of the relayed surface 123H of real-world object or displaysurface 123FS. One or more occlusion planes 150F may have individuallyaddressable occlusion regions 151F, which may be activated to offerocclusion of real-world object or display 123F. These one or moreocclusion planes 150F are relayed by relay system 5090 to relayedposition 150H. A controller 190 may issue coordinated instructions tothe light field display 1001F and the one or more occlusion planes 150Fsimultaneously to arrange for occlusion of the relayed real-worldsurface or display surface 123H by foreground relayed holographicsurface 121H as viewed by observer 1050G and any other observers in theviewing volume 135 of the relayed objects 123H and 121H. Some details ofthe operation of one or more occlusion planes 150 are given above inreference to FIGS. 9B, 9C, and 9D for the configuration of FIG. 9A. Inan embodiment, the one or more occlusion planes 150F are replaced withareal-world occlusion object such as object 155A in FIG. 9E, where theocclusion object may be on a motorized stage which causes the occlusionobject 155A to move 156 in coordination with the movement of relayedholographic object surface 121C. In an embodiment, as shown in FIG. 9E,a controller 190 coordinates instructions to both the light fielddisplay 1001A and the movement of the real-world occlusion object 155A.

FIG. 9G shows light 133D from the surface of display or real-worldobject 123F passing through one or more occlusion planes 150F that maybe comprised of individually-addressable occlusion sites 151F, and thislight 133D being received by a first transmissive reflector relaysubsystem 5030F and relayed along light paths 133E to form first relayedobject surface 123G between the relays. Image light at the first objectrelayed location 123G is relayed from light paths 133E to light paths133F to second object location 123H by the second transmissive reflectorrelay subsystem 5030G. The occlusion plane 150F is relayed to anintermediate virtual plane 150G by the first relay subsystem 5030F, andfrom this position to the second-relayed virtual occlusion plane 150H bythe second relay subsystem 5030G, where the virtual occlusion plane 150Hmay substantially overlap with the relayed holographic image surface121H. The one or more occlusion planes 150F may be configured so anobserver 1050G may not be able to see a portion of the backgroundrelayed object surface 123H behind the foreground relayed holographicobject surface 121H. FIG. 9G provides a side view detail 9003 of opticaldisplay system 9002 that would be observed from observer position 1050F.An image combiner 101F disposed in the light path of the light rays 133Efrom the display or real-world object 123F combines these light rays133E and the light rays 131G forming the holographic object surface121G. The light rays 131G are deflected by the image combiner into lightrays 131H, which travel in the same direction as the light rays 133Efrom the display or real-world object 123F. Both these sets of lightrays are received by the second transmissive reflector relay subsystem5030G. Light rays 131H from the holographic object 121G are relayed tolight rays 131J, forming relayed holographic object surface 121H, whichmay be substantially close or overlapping with the relayed occlusionplane 150H. In the configuration shown in FIG. 9G, the relayedholographic object surface 121H is relayed only once by relay subsystem5030G, which means that relayed holographic surface 121H will have aninverted depth profile relative to projected holographic surface 121G,and so projected holographic surface may have its depth profile invertedby using the optics shown in FIG. 2A or inverting the angular lightfield coordinates (U, V) so the corresponding relayed surface 121H hasthe correct depth. The surface 123FS of display or real-world object123F is relayed twice by depth profile inverting transmissive reflectorrelays 5030F and 5030G so that the corresponding relayed surface 123Hshould appear to an observer 1050G with substantially the same depthprofile as the surface 123FS of display or real-world object 123F. In anembodiment, the first image source 123F shown in FIG. 9G may comprise: a2D display surface, a stereoscopic display surface, an autostereoscopicdisplay surface, a multi-view display surface, the surface or surfacesof a volumetric 3D display, a second light field display surface, thesurface of a real-world object emitting light, or the surface of areal-world object reflecting light. In another embodiment, the secondimage source light field display 1001F in FIG. 9G may comprise: a 2Ddisplay surface, a stereoscopic display surface, an autostereoscopicdisplay surface, a multi-view display surface, the surface or surfacesof a volumetric 3D display, a light field display surface, the surfaceof a real-world object emitting light, or the surface of a real-worldobject reflecting light. In another embodiment, the projectedholographic object 121G may be the relayed surface of a holographicobject.

In the example provided by the illustrated embodiment of FIG. 9G,neither of the transmissive reflector relays 5030F or 5030G is at a45-degree angle with respect to the plane of the display or real-worldobject 123F. One result is that the light rays 133F and 131J projectedfrom the relay system toward an observer 1050G with an optical axis 133Gwhich is not normal to the plane of the display or real-world object123F. An advantage of this configuration is that relay system 9002 maybe placed side-by-side with a similar relay system to generate afield-of-view which is larger than the field-of-view of a single relay9002, which is shown in FIG. 27F below.

While the discussions of FIG. 9A-9G above were made with respect to anembodiment where the relayed holographic image surface is in theforeground and the relayed real-world image surface is in thebackground, the present disclosure also contemplates embodiments wherethe relayed holographic image surface is in the background and therelayed real-world image surface is in the foreground or where both therelayed holographic image surface and the relayed real-world imagesurface are in the foreground or background together. It is to beappreciated that each of these embodiments may be implemented inaccordance with the same principles and operations illustrated byvarious embodiments discussed in the present disclosure.

In this disclosure, there are many permutations of the relayconfigurations that may be implemented in accordance with the principlesdisclosed herein. FIG. 9H is an orthogonal view of some of thecomponents of the optical system 9001 shown in FIG. 9A including relaysystem 5080. The numbering of FIG. 9H applies to FIG. 9I. A first imagesource that may be a display 1001A produces light along paths 131A whichare relayed by first relay subsystem 5030A within relay system 5080 torelayed light paths 131B, forming intermediate virtual display plane1022A, and these light paths are relayed by second relay subsystem 5030Bwithin relay system 5080 to light paths 131C, which form virtual displayplane 1022B. This configuration of the relay system 5080 may also beimplemented with the second relay subsystem 5030B is rotated by 90degrees, which may be desired depending on the requirements of theapplication. FIG. 9I is an orthogonal view of the optical system shownin FIG. 9H, wherein the second relay subsystem 5030B is rotated by 90degrees. The numbering of FIG. 9H applies to FIG. 9I for similarelements. FIG. 9I operates in the same way as FIG. 9H, except that theoutput light 131C in FIG. 9I is relayed in a direction opposite from thedirection of output light 131C in FIG. 9H. The relay system of FIG. 9Hand FIG. 9I may be considered functionally equivalent for the purposesof this disclosure, and no further distinction between the details ofthe configurations shown in FIGS. 9H and 9I will be discussed and bothare referred to herein as the relay system 5080. The same is true formany relay configurations discussed in this disclosure. For example, inrelay 5060 system shown in FIG. 5F, the configuration of the relaysystem 5060 may omit either one of the reflective Fresnel mirrors 1008Aor 1008B and be considered the same relay system 5060. In a similar way,FIG. 9J is an orthogonal view of the optical system shown in FIG. 9H,wherein an image combiner 101 is added between the two relays 5030A and5030B in the relay system 5090 in order to provide a second inputinterface for a second image source operable to define a second imagesurface and produce a set of light rays to be relayed. Light from asecond image source would be sent in a direction perpendicular to theplane of the page and be combined by 101 into light paths which wouldtravel along with light paths 131B (see FIG. 9G). This opticalconfiguration shown in FIG. 9J is a variation of the relay 5090 shown inFIG. 9G but will not be given a separate distinction in this disclosure.

In many of the holographic relay systems, such as relay 5030 shown inFIG. 3A, the holographic object volume centered on the display plane1021 is relayed to a virtual screen plane 1022, which is floating infree space. The distance between the virtual screen plane 1022 and thetransmissive reflector relay 5030 shown in FIG. 3A is determined by thedistance between the transmissive reflector relay 5030 and the displayscreen plane 1021. To achieve the largest distance between a relayedvirtual screen plane and any physical device within a compact design, itmay be advantageous to use an optical folding system in the design. FIG.10A shows an optical folding system 1150 comprised of a plurality ofinternal optical layers, wherein light from the respective image sourceis directed along a plurality of internal passes between internaloptical layers. Such a configuration may be used to increase a distancebetween a relay system and the respective relayed locations. In anembodiment, the optical folding system is comprised of five layers, theoptical folding system receiving light from a display 1101, which may bea LED display, an LCD display, an OLED, or some other type of display.In an embodiment, the internal optical layers comprise first a circularpolarizer comprised of an input polarizer 1111 and a quarter waveretarder 1112, the circular polarizer optically preceding a reflector1113, then a quarter wave retarder 1114, and finally an output polarizer1115. The quarter wave retarder 1114 having an optical axis in a firstdirection. The first quarter wave retarder 1112 has an optical axis in afirst direction, while the second quarter wave retarder 1114 has anoptical axis in a second direction. Light from the display 1101 passesthrough the five or more layers 1111-1115 of the optical fold system1150 in a sequence of three passes with two reflections. FIG. 10Ademonstrates the sequence of reflections and transmissions of light asit travels through the five layers of the optical folding system 1150.The light from the display 1101 passes through the first four layers1111-1114 as part of a first Path 1 2016, reflects from the last layer1115 and passes through layer 1114 as part of a second Path 2 2017, andfinally reflects from layer 1113 and passes through layers 1114 and 1115as part of a third Path 3 2018. Layer 1114 is traversed three times. Inother words, light from an image source is directed between thereflector 1113 and output polarizer 1115 through the quarter waveretarder 1114 in three internal passes. This optical system may bearranged so that layers 1111-1114 are placed together, with minimalspacing between them and far away from layer 1115, as shown in FIG. 10A,so that Path 2 and Path 3 are very close to the length of Path 1,resulting in a total optical path length equal to the length of Paths1-3, which is about three times the length of Path 1 of the optical foldsystem 1150.

In an embodiment, the input polarizer 1111 may include a linearpolarizer, which only transmits light in a first state of linearpolarization, and reflects or absorbs the orthogonal second state oflinear polarization. The quarter wave retarder 1112 of the circularretarder and the quarter wave retarder 1114 may form a pair of quarterwave retarders or quarter wave plates (QWP), where the fast axis angleof the first QWP₁ may be 45 deg relative to the plane of polarization,and the fast axis angle of the second QWP₂ may be −45 deg relative tothe plane of polarization, or vice-versa, so that QWP₂ 1114 may reversethe effect of QWP₁ 1112 on linear-polarized light. The reflector 1113may be a half-mirror reflector formed by a half-transmissive mirror, adielectric mirror, a reflective polarizer, some other reflector. Thereflective polarizer 1115 may reflect a first state of linearpolarization and transmit an orthogonal state of linear polarization, ormay reflect a first state of circular polarization (e.g. left-handcircular polarization LHC) with or without a change in the first stateof circular polarization (e.g. the reflected LHC may be LHC or anorthogonal state of right-hand circular polarization, RHC), and transmita second state of circular polarization (e.g. RHC), orthogonal to thefirst state of circular polarization LHC. The optical fold system 1150may include some other optical layer in some embodiments.

FIG. 10B shows a table which in one embodiment tracks how light from animage source such as display 1101 changes polarization states afterinteracting with each layer of the optical fold system 1150. Lightleaves the display on Path 1, and is filtered by the polarizer layer1111, which may be a linear polarizer, which transmits a first state oflinear polarization L1, and absorbs a second state of linearpolarization L2, orthogonal to the first. This transmitted linearlypolarized light L1 is depicted by the vertical arrow polarization statein the ‘Polariz. State’ row under 1111 and Path 1 in the table of FIG.10B. The quarter wave retarder 1112 converts the linear polarized lightL1 into a circular polarization state LHC, denoted by thecounter-clockwise spiral under 1112 and Path 1 in FIG. 10B. The linearpolarizer 1111 and the quarter wave retarder 1112 are referred to as acircular polarizer because functioning together, they are operable toconvert unpolarized input light into circularly polarized light. Thereflector layer 1113 may be a semitransparent layer, such as ahalf-silvered mirror, and some of the circularly polarized light LHC istransmitted through this layer, labelled as a counter-clockwise spiralunder 1113 and Path 1 in FIG. 10B. The portion of light that is nottransmitted may be reflected back toward the display 1101 in a circularpolarization state RHC, orthogonal to LHC, be converted by layer 1112into a second state of linear polarization L2, orthogonal to the firststate L1, and be absorbed by the polarizer 1111. The LHC polarized lightleaving the reflector 1113 is converted by quarter wave retarder 1114back into linearly polarized light L1 with a first state of linearpolarization L1 (vertical arrow under 1114 and Path 1 in FIG. 10B), andthis first state of linear polarization L1 is reflected by reflectivepolarizer layer 1115 into Path 2 back toward layer 1114 wherein thefirst state of linear polarization L1 is preserved (vertical arrow under1115 and Path 2 in FIG. 10B). The layer 1114 converts this light L1 intotransmitted LHC polarized light, denoted by the counter-clockwise spiralshown in the table of FIG. 10B under 1114 and Path 2. This LHC light isreceived by reflector 1113, and some of this light may be reflected bythe reflector 1113 back toward layer 1114, into Path 3, and this lightmay have a RHC polarization state orthogonal to state LHC as a result ofthe reflection, denoted by the clockwise arrow under 1113 and Path 2 inthe table of FIG. 10B. The quarter wave retarder 1114 coverts this RHCpolarization state into a second state of linear polarization L2,orthogonal to the first state L1, denoted by the horizontal arrow under1114 and Path 3 in the table of FIG. 10B, and this light passes throughthe reflective polarizer layer 1115. In this way, the light from thedisplay has been routed through Path 1, Path 2, and Path 3 beforeleaving the last reflective polarizer layer 1115 of optical foldingsystem 1150.

FIG. 10C is an orthogonal view of a display system comprising an opticalfold system 1160 which offers selective path length extension. Thefolding system 1160 is designed to be placed in the light path of animaging system which increases the path length for a selected area ofincident light rays using a polarization control panel, a polarizationbeam splitter and two planes of reflective surfaces. The polarizationcontrol panel 1123 is a panel that may selectively change the state ofincoming polarization for addressable regions such as 1188 and may be aportion of an LCD panel comprising a plane of liquid crystal. Each planeof reflective surface 1125A and 1125B is paired with a quarter waveretarder plane 1126A and 1126B disposed close to the reflective surface,respectively, in order to create a configuration which will convert alight ray with a first state of polarization into a light ray with asecond state of polarization upon reflection from the reflectivesurface. Light from an object 1121 may be emitted with bothpolarizations, but polarization filter 1122 only allows light paths 1131of a first state of polarization to pass towards the polarizationcontrol panel 1123. In FIG. 10C, light rays of a first polarization aredashed, while light rays of a second polarization orthogonal to thefirst are solid. The light paths 1131 received by the polarizationcontrol panel 1123 may be categorized as a first portion of light rays1131A which are incident on a selected area 1188 of the polarizationcontrol panel and have their first state of polarization changed by thepolarization control panel 1123 into light rays 1132A of a second stateof polarization (solid lines) orthogonal to the first, and a secondportion of light rays 1131B which retain their first state ofpolarization and continue substantially unaffected along light paths1132B (dashed lines). Light rays 1132 leaving the polarization controlpanel include light rays 1132A of the second state of polarization(solid lines) and light rays 1132B of the first state of polarization(dashed lines), which are received by a polarization beam splitter 1130.Light rays 1132B of the first state of polarization (dashed lines) passthrough this polarization beam splitter and exit the optical system1160. Light rays 1132A of the second state of polarization which includelight ray 1133A are deflected by the polarization beam splitter andthese deflected light rays which include light ray 1133B are directedtoward a first paired quarter wave retarder 1126A and reflective surface1125A. Upon reflection from these two planes, the light rays of a secondstate of polarization (solid lines) are converted into light rays with afirst state of polarization (dashed lines), which include light ray1133C, and these light rays pass through the polarization beam splitter1130 toward the second paired quarter wave retarder 1126B and reflectivesurface 1125B. Upon reflection from paired quarter wave retarder 1126Band reflective surface 1125B, the light rays of a first state ofpolarization which include light ray 1133C (dashed lines) are convertedinto light rays with a second state of polarization which include lightray 1133D (solid lines), and these light rays are deflected by thepolarization beam splitter 1130 into output light rays 1133, whichincludes light ray 1133E. Light rays 1132B undeflected by the opticalsystem 1160 in FIG. 10C can be traced back to originate at the sourceobject 1121 at point 1135A, while the light rays 1133 deflected by theswitching region 1188 of the polarization control panel 1123 may betraced back to a common divergence point 1135V. This means that all thelight paths 1131A incident on the polarization control panel 1123 in aselected region 1188 have effectively been path length increased sotheir apparent convergence point 1135V is separated from source point1135A, and the plane of polarization selection 1121 with selectionregion 1188 has been effectively moved back to virtual plane 1121V withvirtual selection region 1188V. An optional output polarization filter1124 may be placed in the optical path of output rays 1132B and 1133 topass only the rays of light 1133 corresponding to the subset of lightrays 1131A from source object 1121 in FIG. 10C that are path-lengthincreased, thereby reflecting or absorbing light rays 1132Bcorresponding to the subset of light rays 1131B that are not path-lengthincreased, thereby providing an optical system which relays the lightpaths passing through a selected occlusion region 1188 to anotherlocation 1188V.

The selective path length extending system 1160 shown in FIG. 10C has aFOV limitation, in that incident light paths 1131 from the object 1121that are at an angle of greater than about 10 degrees from thehorizontal optical axis may not be deflected. FIG. 10D is an orthogonalview of an optical fold system 1170 which increases the path length fora selected region of light rays in a low refractive index n˜1 medium1161 using a polarization beam splitter embedded in a medium of highrefractive index n>1 material 1162, and two planes of reflectivesurfaces to increase the field of view of the optical system shown inFIG. 10C. The high refractive index material 1162 within the nearprism-shaped boundary 1144 bends incident light towards the opticalaxis, thus increasing the acceptance angle of incident light rays.Otherwise, the principle of operation of selective path length expander1170 is similar in operation to selective path length expander 1160.Incident light rays 1151A, 1152A, 1156A, and 1157A of a firstpolarization (dashed lines) may be produced by a source 1121 and apolarization filter 1122, where 1121 and 1122 are not part of theselective optical fold system 1170. These light rays are received by apolarization control panel 1143 which may selectively switch onepolarization state to another in addressable regions such as region 1188and may be a portion of an LC panel. Light rays 1151A pass through thisselected region, and are converted into a second state of polarization1151B (solid lines) which are deflected by the polarization beamsplitter 1149 into light rays 1151C, which reflect from a first pairedquarter wave retarder 1146A and reflective surface 1145A into lightpaths 1151D, switching polarization state into the first polarizationstate (dashed lines), and passing through the polarization beam splitter1149. Upon reflection from the second paired quarter wave retarder 1146Band reflective surface 1145B, light paths 1151D of the firstpolarization state are converted into light paths 1151E of a secondpolarization state (solid lines), which deflect from the polarizationbeam splitter 1149 and exit the optical system 1170 as light paths1151F. Similarly, incident light paths 1152A follow a similar path andexit the optical system 1170 as light paths 1152F. Light 1156A and 1157Aincident on areas of the polarization control panel which are notselected may not switch polarization state, but of this group of lightrays the ones that are incident at an angle to the normal to the planeof the boundary 1144 are deflected toward the horizontal optical axisinto light paths 1156B and 1157B, respectively, upon entering the regionof a higher index of refraction 1162. Upon leaving the high-index medium1162, the light paths 1156B and 1157B that are at an angle with respectto the horizontal optical axis are deflected away from the optical axisin accordance with Snell's law into light paths 1156C and 1157C.Although it is not shown in the optical system 1170, the light rays1151A and 1152A that are selected by the polarization control plane anddeflected by the polarization beam splitter 1149 have a virtualconvergence point to the left of the source object plane 1121 much likeconvergence point 1135V in FIG. 10C, and the selective polarizationcontrol plane may have a corresponding virtual plane between thisvirtual convergence point and the source object 1121, similar to plane1121V in FIG. 10C. As in FIG. 10C, an optional polarization filter 1124may be placed in the optical path of output rays 1151F, 1152F, 1156C,and 1157C to pass only light rays 1151F and 1152F corresponding to thelight rays 1151A and 1152A from source object 1121 which are path lengthincreased, thereby providing an optical system which relays the lightpaths passing through a selected occlusion region 1188 to anotherlocation (e.g. similar to 1188V in FIG. 10C).

FIGS. 11A, 11B, and 11C show embodiments of an optical system comprisinga first input interface configured to receive light along a first set oflight paths from a first image source, wherein the light from the firstimage source is operable to define a first image surface; and a secondinput interface configured to receive light along a second set of lightpaths from a second image source, wherein the light from the secondimage source is operable to define a second image surface; and a firstrelay system configured to receive combined image light from the opticalcombining system and relay the received light to relayed locations in aviewing volume thereby defining first and second relayed image surfacescorresponding to the first and second image surfaces respectively;wherein at least one of the first and second image sources comprises alight field display, and the first set of light paths are determinedaccording to a four-dimensional function defined by the light fielddisplay such that each projected light path has a set of spatialcoordinates and angular coordinates in a first four-dimensionalcoordinate system. FIG. 11A shows a general relay system 5000 whichreverses the depth profile of surfaces it relays, while FIG. 11B shows ageneral relay system 5001 which preserves the depth profile of thesurfaces it relays. FIG. 11C shows a slightly different configuration ofFIG. 11B.

FIG. 11A shows an example of a display system comprising an opticalcombining system 101 and a first relay system 5000 which reverses thedepth profiles of objects that it relays. The numbering of FIG. 9A isused in FIG. 11A for similar elements. The relay system 5000 may berelay 5010 shown in FIG. 1A, relay system 5020 shown in FIG. 1B, therelay system 5030 shown in FIG. 3A, or any other relay system whichperforms depth reversal. The relay system 5000 may also be relay system5100 to be introduced in FIGS. 20 and 22 below. In FIG. 11A, light fielddisplay 1001A projects light ray groups 131A and 132A to produceholographic surfaces 121A and 122A, respectively. The light rays 131Aand 132A are combined with light rays 133Y from the surface 123AS of areal-world object 123A by an image combiner 101, wherein the imagecombiner 101 deflects the light rays 133Y into light rays 133A so theyare travelling in the same direction with the portion of light rays 131Aand 132A which pass through 101. These combined light rays 131A, 132A,and 133A are received by the relay system 5000 and relayed to light rays131B, 132B, and 133B. Light rays 131B and 132B form relayed holographicobject surfaces 121B, 122B around virtual relayed screen plane 1022A,respectively, while light rays 133B form the relayed surface 123BS ofreal-world object 123A. The relayed surfaces 121B, 122B, and 123BS havebeen relayed to a viewing volume defined by boundary 1060 and viewableby observer 1050. The viewing volume boundary 1060 is illustrated inFIGS. 11A-11J to indicate the location where relayed surfaces may beseen fully within the field of view of the display. An observer 1050will view the relayed surfaces 121B, 122B, and 123BS from within theviewing volume boundary 1060. This boundary is not shown in otherfigures in this disclosure. Notice that the relayed holographic surfaces121B and 122B are depth reversed from their projected holographicsurfaces 121A and 122A, respectively, while the surface 123BS ofreal-world object 123B is also depth reversed compared to the surface123AS of the real-world object 123A. In an embodiment, a holographicsurface 121A/122A is formed by light paths 131A/132A projected from thelight field display 1001A and has a first projected depth profile, andthe first relayed image surface 121B/122B comprises a relayedholographic surface with a first relayed depth profile that is differentfrom the first projected depth profile. In an embodiment, the lightfield display comprises a controller 190 configured to issueinstructions for accounting for the difference between the firstprojected depth profile and the first relayed depth profile by operatingthe light field display 1001A to output projected light such that thefirst relayed depth profile of the first relayed image surface 121B/122Bis the depth profile intended for a viewer 1050. In another embodiment,the relayed locations of the first relayed image surface 121B/122B aredetermined according to a second 4D function defined by the relaysystem, such that the received light paths 131A/132A and 133A from thefirst and second image sources, respectively, are relayed along relayedlight paths 131B/132B and 133B each having a set of spatial coordinatesand angular coordinates in a second 4D coordinate system defined withrespect to a first virtual display plane 1022A, wherein the light fielddisplay 1001A comprises a controller configured to issue instructionsfor accounting for the second 4D function by operating the light fielddisplay 1001A to output projected light according to the first 4Dfunction such that the positional coordinates and angular coordinates inthe second 4D coordinate system for each of the set of relayed lightpaths 131B/132B respectively, allow the first relayed image surface121B/122B to be presented to a viewer as intended. One or more occlusionlayers 151, 152, and 153 with individually-addressable regions such as188 may be disposed in the optical path of light rays 133Y from thereal-world object 123A to offer occlusion of the real-world object 123Amuch the same way as pictured in FIGS. 9B, 9C and 9D. Optional opticalpath folding system 1150 shown in FIG. 10A-B, 1160 shown in FIG. 10C, or1170 shown in FIG. 10D may be disposed in the path of light 131A and132A from the light field display 1021A or the light 133Y from thereal-world object 123A in order to increase the relative path length ofthese light rays, causing the corresponding surfaces produced by thoselight rays to be relayed further from the relay 5000. For example, if apath length extender 1150, 1160, or 1170 is disposed in the path oflight rays 131A and 132A, then the relayed holographic surfaces 121B and122B as well as the virtual relayed screen plane 1022A will all berelayed closer to the observer 1050 and further from the relay 5000. Asshown above, a selective optical fold system 1160 shown in FIG. 10C orselective optical fold system 1170 shown in FIG. 10D may be used toselectively extend the path lengths of a first group of light rays 131Aforming holographic surface 121A without affecting the second group oflight rays 132A forming holographic surface 122A, and vice-versa. As anexample, activating an optical fold system in the path of light rays131A from projected surface 121A would move the corresponding relayedsurface 121B closer to observer 1050. In an embodiment, the displaysystem shown in FIG. 11A may comprise a controller 190 which issuescoordinated display instructions to the light field display 1001A,configuration instructions to the occlusion layers of an occlusionsystem 150, and configuration instructions for a selective optical foldsystem 1160 or 1170.

In this disclosure, sometimes no distinction is made between a relayedobject and a relayed surface. In FIG. 11A, the projected holographicobjects 121A and 122A are surfaces which are relayed by relay system5000 to relayed holographic surfaces 121B and 122B, respectively. Theprojected holographic object surfaces 121A and 122A, as well as therelayed holographic object surfaces may be referred to as ‘projectedholographic object surfaces’ or ‘projected holographic objects’ or even‘holographic objects’ equally in this disclosure. The correspondingrelayed holographic surfaces 121B and 122B may be referred to as‘relayed holographic surfaces’ or ‘relayed holographic objects’.Similarly, in FIG. 11A, a real-world object 123A has a surface 123ASwhich reflects or emits light, and the light from this surface 123AS isrelayed to relayed surface 123BS by relay system 5000. This disclosuremay use the equivalent description of a ‘real-world object’ beingrelayed to ‘relayed real-world object’ or ‘relayed image of real-worldobject’, without mention of surfaces—sometimes the real-world object123A or the relayed real-world object 123B will be shown without anyseparate mention of surfaces. Also, the image source for a holographicsurface is a light field display, which projects light which convergesat the surface of a holographic object and leaves this surface just asif a real object were there emitting or reflecting light. In thisexample, the surface of a holographic object is a true location ofconverged light. However, the image surfaces produced by other types ofimage sources, such as some stereoscopic, autostereoscopic displays, orhorizontal parallax only (HPO) multi-view displays are operable todefine perceived image surfaces even though the viewer may be focusinghis or her eyes at the display screen when observing these perceivedsurfaces. In these instances, the relay will relay the light raysforming a perceived image surface to a perceived relayed image surfaceat another location that may be observed by a viewer.

The field-of-view of a light field display 1001A may be more limitedthan angular range of light leaving a real-world object 123A. In somecircumstances, in order to allow the observer 1050 to see a consistentfield-of-view for both the relayed holographic object surfaces 121B and122B as well as the relayed image surface 123B of real-world object123A, and to also reduce stray light that may enter the relay system5000, an angular filter 124 may be placed in front of the real-worldobject 123A in order to absorb or reflect away light that is beyond anintended field of view for the observer or the optical system. In theembodiment shown in FIG. 11A, the angular filter 124 absorbs rays oflight 133R from the real-world object 123A that have an angle withrespect to the normal to the surface of the angular filter that exceedsa threshold value. In all following example figures showing light fielddisplay systems, which combine relayed images of real-world objects withrelayed holographic objects, an angular filter 124 may be used in frontof the real-world object 123A, whether or not it is shown in the figure.

FIG. 11B is an example of a display system comprising the sameconfiguration of FIG. 11A, except that the relay system 5001 preservesthe depth profile of the image surface it relays. The numbering of FIG.11A is used in FIG. 11B. The relay system 5001 in FIG. 11B may be relaysystem 5040 shown in FIGS. 4C and 5D, relay system 5050 shown in FIG.5E, relay system 5060 shown in FIG. 5F, relay system 5070 shown in FIG.4E, relay system 5080 shown in FIG. 9A, relay system 5090 shown in FIG.9G, or any other relay system that doesn't reverse depth. The relaysystem 5001 may be relay system 5110 to be introduced in FIG. 25A, orrelay system 5120 to be introduced in FIG. 25B below. The light fielddisplay 1001A in FIG. 11B projects depth reversed holographic objectsurface 121AR in place of 121A shown in FIG. 11A, and 122AR in place of122A shown in FIG. 11A so the corresponding relayed holographic objectsurfaces 121B and 122B are the same as shown in FIG. 11A. Note that inFIG. 11B, the projected holographic surfaces 121AR and 122AR have adepth profile relative to display plane 1021A which is the same as thedepth profile of their respective relayed holographic surfaces 121B and122B relative to the relayed display plane 1022A. Relayed real-worldobject surface 123BS has a depth profile which is also the same asreal-world object 123A depth profile 123AS, and since relayed surface123BS is further from the virtual screen plane 1022A then relayedholographic surfaces 121B and 122B, the corresponding real-world object123A must also be located at a greater distance (optical path length)from the image combiner 101 than projected holographic object surfaces121AR and 122AR. In an embodiment shown in FIG. 11B, a relay system 5001is configured to relay the relayed image surface 123B of the real-worldobject 123A to the relayed locations that define the respective relayedimage surface 123B of the real-world object in the viewing volumedefined by boundary 1060 and viewable by observer 1050 such that therespective relayed image surface 123B of the real-world object in theviewing volume has a depth profile that is substantially the same as thedepth profile of the surface of the real-world object 123A.

In an embodiment, the relay system of FIG. 11A may further include anocclusion system configured according to any embodiment described in thepresent disclosure, include the occlusion system 150 discussed abovewith respect to FIGS. 9A-9D. The occlusion system may be comprised of areal-world occlusion object 155A shown in FIGS. 9E and 9F, which will beshown below in FIG. 11C. In addition, the controller 190 may senddisplay instructions to the light field display 1001A as well as theocclusion system 150, which as discussed above, may include one or moreocclusion planes 151, 152, and 153. A controller 190 may issue displayinstructions to the light field 1001A and simultaneously issue occlusioninstructions to the occlusion layers 151, 152, and 153 in order tocorrectly occlude the relayed surface of the real-world object 123BSbehind one or more of the relayed holographic surfaces 121B and 122B asviewed by a viewer 1050 anywhere in the field of view of the relayedobjects 121B, 122B, and 123B. In subsequent diagrams that appear in thisdisclosure, the controller 190 may not be shown as connected to theocclusion system 150, but it should be assumed that the controller maybe connected to the occlusion system 150 as well as the image source1001A in the system.

FIG. 11C is the display system of FIG. 11B with the occlusion system 150replaced by a real-world occlusion object 155A, and an enclosure whichblocks ambient light from entering the relay system 5001. The numberingof FIG. 11B is used in FIG. 11C. The real-world occlusion object 155Awas presented in reference to FIG. 9E, and the ambient light rejectionenclosure 1080 is presented in reference to FIGS. 5G and 5H above. Theocclusion object 155A blocks unwanted light rays from the real-worldobject 123A. The real-world occlusion object 155A may be similar inshape or profile to at least one projected holographic object 121AR andmay be painted or coated with a light absorbing material such as matteblack paint. In FIG. 11C, the real-world occlusion object 155A has beenpositioned so that it is equidistant from the image combiner 101 as theprojected holographic object 121AR and thus has an equal optical pathlength to the relay system 5001 as holographic object 121AR. Because ofthis, if the real-world occlusion object 155A were reflective oremissive, the surface of 155A would be relayed to relayed surface 155Bby the relay system 5100 so that it coincides at substantially the samelocation as the relayed surface 121B of the projected holographic objectsurface 121AR. As shown above in reference to FIG. 5G, some of the lightrays 133YS from the surface 123AS of real-world object 123A are blockedby the real-world occlusion object 155A (dashed lines). The entiredistribution of light rays from surface 123AS, including 133YS and 133Ythat are unobstructed by 155A is relayed by the relay system 5001 intolight rays 133YSR and 133B, and these light rays offer occlusion ofrelayed surface 123BS of real-world object 123A by relayed holographicobject 121B for substantially all angles of relayed light from surface123AS, given the same relative placement of relayed holographic objectsurface 121B to relayed real-world object surface 123B compared to theplacement of real-world occlusion object 155A to real-world objectsurface 123AS, as well as substantially the same dimensions of thereal-world occlusion object 155A to relayed holographic object surface121B. For reference, FIG. 9F shows the effect of the real-worldocclusion object 155A shown in FIG. 9E on the relayed real-world objectimage surface 123C, as viewed by observer positions 1050A, 1050B, and1050C shown in FIG. 9E. In summary, FIG. 11C shows that in a displaysystem in which the light from a projected holographic surface 121AR anda real-world object surface 123A are combined and relayed, then areal-world occlusion object 155A with the same dimensions as thedimensions of the relayed holographic object surface 121B may be placedin a location which blocks a portion of the light from the real-worldobject 123A such that the relayed holographic object surface 121B andthe relayed surface of real-world occlusion object 155B are coincident,the real-world occlusion object 155A offering occlusion of the relayedreal-world object surface 123B behind the relayed holographic objectsurface for all viewers 1050 within the FOV of the relayed objectsurfaces 121B and 123B. In an embodiment, the real-world occlusionobject 155A has its location controlled by a motorized positioning stage(not shown), and 155A can be moved 156 in coordination with the movementof a projected holographic object 121A so that the relayed position 155Bof relayed occlusion object 155A continually coincides with the positionof a relayed holographic object surface 121B. A controller 190 maysimultaneously issue display instructions to the light field display1001A as well as issue commands to a motion controller in order todirect coordinated movement 156 of the real-world occlusion object 155Aas well as movement of a projected holographic object 121AR. While therelay 5001 shown in FIG. 11C does not invert the depth profile ofrelayed objects 121AR, 122AR, and 123A, it is possible to use anocclusion object in a relay which does invert depth such as relay 5000in FIG. 11A. In this case, the real-world object 123A could be replacedby a relayed real-world object with reversed depth. To arrange this, thereal-world occlusion object 155A and a real-world object copy of 123Amay have the same relative placement of 155A and 123A shown in FIG. 11C,but the real-world object copy of 123A would be relayed to the location123A shown in FIG. 11C using a relay which inverts depth, such as atransmissive reflector relay 5030. Such a configuration will be shown inthe display system 1400 in FIG. 14A presented below.

Many of the display systems in this disclosure are designed to relaylight from one or more light sources through a relay system and to anobserver. For the purposes of avoiding unwanted scattering andreflection within these display systems, it is best to avoid directinglight into the display system in a direction opposite to the directionof the light being relayed and seen by one or more viewers. It is notalways possible to keep the viewing area for relayed objects presentedby a display system in the dark. FIG. 11C shows the display system ofFIG. 11B confined to a light blocking enclosure or portion of anenclosure 1080 with a polarization filter 1081 used as a window in thepath of relayed light paths in order to reject ambient environmentallight. This ambient light rejection system comprised of enclosure 1080and polarization filters 1081 and 1082 is discussed above with respectto FIGS. 5G and 5H for the case when relay 5001 is relay 5060. Thepolarization filter 1081 is placed in the path of relayed light paths131B and 132B forming the surfaces 121B and 122B of relayed holographicobjects, respectively, as well as relayed light paths 133B forming therelayed surface 123BS of a real-world object. The window 1081 may onlypass the portion of these relayed light paths 131B, 132B, and 133B thatare in a first state of polarization, while absorbing or reflecting theportion of these relayed light paths that is in a second state ofpolarization. The environmental light source 1085 produces light of twopolarizations 1091, but a light source polarization filter 1082 onlyallows light 1092 of a second state of polarization to pass through andilluminate the environment around the display system, and this lightwill not pass through the polarization filter window 1081 of the displaysystem and reflect or scatter from elements within the relay 5001 or anyother components in display system in FIG. 11C. In an embodiment, apolarized light source 1085 may be used without a light sourcepolarization filter 1082. It should be appreciated that the ambientlight rejection system formed by ambient light polarization filter 1082,the light blocking enclosure 1080, and the display system polarizationfilter window may be used for any of the display systems with relayspresented in this disclosure.

In FIGS. 11A-C, the optical combining system 101 may include a firstinput interface configured to receive light along a first set of lightpaths (e.g. 131A) from a first image source which is the surface 1021Aof light field display 1001A wherein the light from the first imagesource is operable to define a first image surface (e.g. 121A in FIG.11A, 121AR in FIGS. 11B and 11C); and a second input interfaceconfigured to receive light along a second set of light paths (e.g.133Y) from a second image source (e.g. emissive or reflective surface123AS of real-world object 123A), wherein the light from the secondimage source is operable to define a second image surface (e.g. 123AS).In an embodiment, the first image source 1001A comprises the surface1021A of a light field display 1001A as shown in FIG. 11A operable todefine a holographic first image surface (e.g. 121A in FIG. 11A, 121ARin FIG. 11B), and the first set of light paths (e.g. 131A) of the lightfield display 1001A image source is determined according to afour-dimensional function defined by the light field display 1001A suchthat each projected light path (e.g. 131A) has a set of spatialcoordinates and angular coordinates in a first four-dimensionalcoordinate system defined with respect to a light field display screenplane 1021A. The first image surface of the light field display 1001Amay include a holographic surface, such as holographic surfaces 121A and122A in FIG. 11A, and 121AR and 122AR in FIG. 11B.

In an embodiment, the second image source 123A may include the surfaceof a 2D display, a stereoscopic display surface, an autostereoscopicdisplay surface, a multi-view display surface including a multi-viewdisplay surface in one axis (e.g. the surface of a horizontal parallaxonly or HPO display such as a lenticular display), the surface orsurfaces of a volumetric 3D display, a second light field displaysurface, the surface of real-world object emitting light, or the surfaceof a real-world object reflecting light. Correspondingly, the imagesurface of the second image source may include an image surfaceprojected from a 2D display surface, an image surface projected from astereoscopic display surface, an image surface projected from anautostereoscopic display surface, an image surface projected from amulti-view display surface, an image surface of a volumetric 3D display,a surface of a holographic object formed by light paths projected from asecond light field display, a surface of a real-world object, or arelayed image of the surface of the real-world object. In an embodiment,the first relay system 5000 or 5001 may be configured to receivecombined image light from the optical combining system 101 and relay thereceived light to relayed locations in a viewing volume defined byboundary 1060 and viewable by observer 1050, whereby first and secondrelayed image surfaces 121B/122B and 123B in FIGS. 11A-C are observableat the respective relayed locations. The image source for a holographicobject is a light field display surface, which projects light whichconverges at the surface of a holographic object and leaves this surfacejust as if a real object were there emitting or reflecting light. Inthis example, the surface of a holographic object is a true location ofconverged light. However, the image surfaces produced by other types ofimage sources, such as some stereoscopic, autostereoscopic displays, orhorizontal parallax only (HPO) multi-view displays are operable todefine perceived image surfaces even though the viewer may be focusinghis or her eyes at the display screen when observing these perceivedsurfaces. In these instances, the relay will relay the light raysforming a perceived image surface to a perceived relayed image surfaceat another location that may be observed by a viewer.

Many variations of the configuration shown in FIG. 11A-C are possible.The occlusion system may comprise an occlusion system opticallypreceding at least one of the first and second input interface (e.g. onlight path 133Y in FIG. 11A), the occlusion system configured to occludea portion of at least one of the first and second image surfaces (e.g.surface 123A in FIGS. 11A-C), wherein the occluded portion correspondsto a relayed occluded portion of at least one of the first and secondrelayed image surfaces (e.g. occluded portion 189 of relayed imagesurface 123BS in FIGS. 11A-B), the relayed occluded portion (e.g. 189 inFIGS. 11A-B) being observable as being occluded by the other one of thefirst and second relayed image surfaces (e.g. relayed image 121B inFIGS. 11A-B). In an embodiment, the occlusion system comprises at leastone occlusion layer (e.g. layers 151, 152, and 153 of occlusion system150 in FIG. 11A). In an embodiment, the occlusion layer comprises one ormore individually addressable elements (e.g. 188 in FIGS. 11A-B). Theone or more individually addressable elements may comprise occlusionsites configured to block a portion of incident light or parallaxbarriers. In an embodiment, the one or more occlusion layers withindividually addressable elements comprises one or more transparent LEDpanels, transparent OLED panels, LC panels, or other panels operable toselectively occlude light.

In an embodiment the first relayed image surface 121B in FIGS. 11A-Bcomprises a foreground surface in front of the second relayed imagesurface 123B comprising a background surface, and the at least oneocclusion layer is located in front of second image source 123A and isoperable to define an occlusion region 188 having a size and shapescaled to that of the foreground surface 121B so that an occludedportion 189 of the background surface 123B cannot be observed behind theforeground surface 121B. In an embodiment, a distance between the atleast one occlusion layer 152 and the second image surface source 123ASis substantially equal to a distance between the foreground relayedsurface 121B and the background relayed surface 123B. In an embodiment,the occlusion region 188 defined by the at least one occlusion layer isrelayed to the viewing volume defined by boundary 1060 to substantiallycoincide with the foreground surface 121B. In an embodiment, the opticalsystem further comprises a controller operable to coordinate a movementof the occlusion region 188 with a movement of an image surface121B/122B in the viewing volume defined by boundary 1060. In anembodiment, the movement of the occlusion region in the at least oneocclusion layer 152 in FIG. 11A is effected at least in part bymodulating individually addressable elements 188 in FIG. 11A in the atleast one occlusion layer.

In an embodiment, the occlusion system may be provided by a real-worldocclusion object (155A in FIG. 11C), and this occlusion object may bemotorized so it's relayed position (155B in FIG. 11C) may stay insynchronization with the relayed image surface (121B in FIG. 11C). In anembodiment, and referencing FIG. 11C, the first relayed image surface121B comprises a foreground surface in front of the second relayed imagesurface 123B comprising a background surface, and wherein the at leastone occlusion object 155A is located in front of the second image source123A, and the size and shape of the at least one occlusion object 155Ais scaled to that of the foreground surface 121B in the viewing volumedefined by boundary 1060 so that an occluded portion of the backgroundsurface 123BS cannot be observed behind the foreground surface 121B. Inan embodiment, and referencing FIG. 11C, a distance between the at leastone occlusion object 155A and the second image surface source 123A issubstantially equal to a distance between the foreground 121B andbackground 123B relayed surfaces. In another embodiment, and referencingFIG. 11C, an occlusion region defined by the at least one occlusionobject 155A is relayed to the viewing volume defined by boundary 1060 to155B to substantially coincide with the foreground surface. In anembodiment, the at least one occlusion object 155A is motorized so itmay be moved 156. In another embodiment, the optical system furthercomprises a controller 190 operable to coordinate a movement 156 of theat least one occlusion object 155A with a movement of a relayed imagesurface 121B, 122B, or 123B in the viewing volume defined by boundary1060. In an embodiment, a first relayed image surface 121B/122B in FIGS.11A-C is observable in the foreground, while a second relayed imagesurface 123B in FIGS. 11A-C is observable in the background. In anotherembodiment, the first relayed image surface could be observable in abackground, and the second relayed image surface could be observable inthe foreground. In still another embodiment, the first and secondrelayed image surfaces may be both observable in a foreground or abackground. In an embodiment shown in FIG. 11B, wherein the relay systemdoes not reverse the depth profile of a relayed object surface, a relaysystem is configured to relay the relayed image surface 123B of thereal-world object 123A to the relayed locations that define therespective relayed image surface 123B of the real-world object in theviewing volume defined by boundary 1060 such that the respective relayedimage surface 123B of the real-world object in the viewing volume has adepth profile that is substantially the same as the depth profile of thesurface of the real-world object 123A.

In an embodiment, there may be an optical fold system opticallypreceding at least one of the first and second interfaces of the opticalcombining system 101 (in the path of light from the holographic display1001A or in the path of light from the real-world object 123A in FIGS.11A-C). Alternatively, in FIG. 11A, the optical fold system 1150 may beplaced: between the optical combining system 101 and the relay system5000 (after the light 131A and 132A from the holographic objects hasbeen combined with the light 133Y from the real-world object 123A);between the relay system 5000 and the observer 1050, or in some otherlocation in an optical path of the system. An optical fold system 1150may be used to extend the path lengths of light from either first source1001A or second source 123A. As shown above, a selective optical foldsystem (selective path length extender) 1160 shown in FIG. 10C orselective optical fold system 1170 shown in FIG. 10D may be used toselectively extend the path lengths of a first group of light rays 131Ain FIG. 11C forming holographic surface 121AR without affecting thesecond group of light rays 132A forming holographic surface 122AR, andvice-versa. As an example, activating an optical fold system in the pathof light rays 131A from projected surface 121AR would move thecorresponding relayed surface 121B closer to observer 1050. In anembodiment, the display system shown in FIG. 11C may comprise acontroller 190 which issues coordinated display instructions to thelight field display 1001A, configuration instructions to motioncontrollers responsible for movement 156 of occlusion object 155A, andconfiguration instructions for a selective optical fold system 1160 or1170.

In an embodiment the optical display system of FIGS. 11A-C may furthercomprise an optical fold system optically preceding one of the first andsecond interfaces of the relay 5000 or 5001. These optional optical foldsystems are labelled 1150, 1160, or 1170 located in the paths of light133A from first image source 123A or located in the light paths 131A and132A from second image source 1001A in FIGS. 11A-C. Optical fold system1150 is described in detail above in reference to FIGS. 10A-B, whileselective optical fold systems 1160 and 1170 are described above indetail in reference to FIGS. 10C and 10D, respectively. In anembodiment, the optical fold system 1150, 1160, or 1170 comprises aplurality of internal optical layers, and light from the respectiveimage source 1001A or 123A is directed along a plurality of internalpasses between internal optical layers thereby increasing an opticalpath distance between the relay subsystem and image surface locations inthe viewing volume defined by boundary 1060. In an embodiment, in FIGS.11A-C, one image source comprises the light field display 1001A, and theoptical fold system is located in the path of the light 131A and 132Afrom the light field display to increase the optical path lengthdistance between respective image surface locations 121B/122B in theviewing volume defined by boundary 1060 and the relay system 5000 or5001. In an embodiment, referencing FIGS. 11A-C, one image sourcecomprises the light field display 1001A, and the optical fold system islocated in the path of the second image source 123A to increase theoptical path length distance between respective image surface locationssuch as 123B in the viewing volume defined by boundary 1060 and therelay system 5000 or 5001. In another embodiment, the optical systemshown in FIG. 11C may further comprise an optical fold system opticallyfollowing at least one of the first and second interfaces of the relaysystem, within the internal layers of the relay system 5001 or on theoutput of the relay system 5001 in the path of light rays 131B, 132B,and 133B. In an embodiment, the optical systems shown in FIG. 11A-C havean environmental light rejection system as shown in FIG. 11C whichcomprises an enclosure (e.g. 1080 in FIG. 11C) that partially enclosesthe relay system and a window comprising a polarization filter (e.g.1081 in FIG. 11C). In a further embodiment, the polarization filter isoperable to block ambient light having a first polarization state. Theambient light has may have a first polarization state and is provided bya light source comprising a polarization output filter configured toallow light only of the first polarization state to pass through (e.g.light source 1085 being filtered by polarization output filter 1082 inFIG. 11C).

The relay system 5001 in FIG. 11B may be configured like relay system5080 in FIG. 9A or relay system 5090 in FIG. 9G such that the real-worldobject 123A may be relayed twice possibly for the purpose of solvingdepth reversal. In some configurations, the relay system 5001 mayintroduce magnification changes of the relayed holographic objects orreal-world objects, like relay 5040 in FIG. 5D, 5050 in FIG. 5E, or 5060in FIG. 5F. In other configurations, the relay 5001 may introduce u-vangular coordinate remapping for light rays, as described above for thecurved surface relays 5040 in FIG. 5D and 5050 in FIG. 5E, or theFresnel mirrors of relay 5060 in FIG. 5F. The relay may introduce a 90degree rotation between the light field display plane 1021A and therelayed virtual display plane 1022A, a 180 degree rotation, or, inanother embodiment, no rotation in a configuration where the relay isin-line with the light field display 1001A and the observer, describedbelow. In some configurations, there is substantial distance between thefirst relayed image surface 121B/122B of the light field display 1001Aand the second relayed image surface 123B of the real-world object 123A.In another embodiment, the relay system 5000 or 5001 may relay only theholographic object surfaces 121A/122A in FIG. 11A and 121AR/122AR inFIG. 11B, and merely transmit the light from the real-world objectwithout relaying it, or, conversely, the relay may relay only the imagesurface 123A from the real-world object and merely transmit the lightfrom the respective holographic object surfaces 121A/122A in FIG. 11Aand 121AR/122AR in FIG. 11B without relaying the holographic objectsurfaces. Examples of many of these configurations are given below.

The next two figures FIGS. 11D and 11E illustrate optical systemscomprising: an optical combining system comprising a first inputinterface configured to receive light along a first set of light pathsfrom a first image source, wherein the light from the first image sourceis operable to define a first image surface; a second input interfaceconfigured to receive light along a second set of light paths from asecond image source, wherein the light from the second image source isoperable to define a second image surface; a relay system configured toreceive combined light from the optical combining system and relay thereceived light to relayed locations in a viewing volume defined byboundary 1060, whereby first and second relayed image surfaces areobservable at the respective relayed locations; and an occlusion systemconfigured to occlude a portion of light from at least one of the firstand second image sources. In these optical systems, neither the firstimage source nor the second image source is required to be a light fielddisplay, but otherwise these optical systems are like the opticalsystems shown in FIGS. 11A-C.

FIG. 11D is the display system of FIG. 11A with the first image sourcelight field display 1001A replaced by display 990A with display surface991A. The numbering of FIG. 11A is used in FIG. 11D. Light rays 131G and132G from the first image source display 990A with surface 991A arerelayed to light paths 131H and 132H, respectively, and are focused onrelayed virtual display plane 992A. Real-world object 123B is relayed tothe same place as shown in FIG. 11A. Sites 188 on occlusion planes151-153 may be activated to block out some of the light from real-worldobject 123A, so that portions of the relayed image 123B of thereal-world object cannot be seen behind relayed images on the virtualdisplay plane 992A. The controller 190 may issue instructions to theocclusion system 150 as well as the first image source 990A. In analternate configuration, light rays 133Y may be blocked using areal-world occlusion object like 155A shown in FIG. 11C, and thisocclusion object may be moved using one or more motorized stages asdirected by the controller 190. In an embodiment, while the first andsecond image sources in FIG. 11D are a display 990A and a real-worldobject 123A, the first and second image sources can each be any of: a 2Ddisplay surface, a stereoscopic display surface, an autostereoscopicdisplay surface, a multi-view display surface which may be the surfaceof a horizontal parallax-only multi-view display such as a lenticulardisplay, the surface or surfaces of a volumetric 3D display, the surfaceof a real-world object emitting light, or the surface of a real-worldobject reflecting light. The light from each of the first and secondimage source is operable to define a corresponding image surface whichmay be any of: an image surface projected from a 2D display surface, animage surface projected from a stereoscopic display surface, an imagesurface projected from an autostereoscopic display surface, an imagesurface projected from a multi-view display surface, the image surfaceof a volumetric 3D display, the surface of a holographic object formedby light paths projected from a light field display, a surface of areal-world object, or a relayed image of the surface of a real-worldobject. In an embodiment, the depth profile reversing relay 5000 in FIG.11D may be replaced with another relay 5001 introduced in FIG. 11B whichdoes not perform depth reversal, resulting in projected image surfacesdefined by first and second image sources being relayed to relayed imagesurfaces with different depth profiles than the projected imagesurfaces.

In another embodiment, and as a further configuration option of therelay system shown in FIG. 11A, the real-world object 123A in FIG. 11Dmay be instead may be a second display. FIG. 11E is the display systemof FIG. 11A with both the light field display 1001A and the real-worldobject 123A both replaced by displays 990A and 992A, possibly ofdifferent types. In FIG. 11E, display surface 991A of display 990A anddisplay surface 993A of display 992A may each be a 2D display surface, astereoscopic display surface, an autostereoscopic display surface, amulti-view display surface, the surface or surfaces of a volumetric 3Ddisplay, a light field display surface, the surface of a real-worldobject emitting light, or the surface of a real-world object reflectinglight. Some of the numbering of FIG. 11D is used in FIG. 11E. Lightpaths 131G and 132G from display 990A are relayed to light paths 131Hand 132H, respectively, forming a focused first virtual relayed imageplane 992A. Light paths 996A from display 993A are deflected by theimage combiner 101 into light paths 996B, the light paths 996B receivedby relay 5000 and relayed to light paths 996C which converge on a secondrelayed virtual image plane 994A. Light paths 996R at a high angle maybe rejected by an angle filter 124. For observer 1050, virtual relayedimage plane 992A is in front of relayed image plane 994A, and soocclusion regions 188 on the one or more occlusion planes 151-153 may beactivated in order to block portions of light 189 from the backgroundrelayed image plane 994A from being seen behind foreground images on theforeground relayed image plane 992A. The controller 192 may be connectedto the occlusion system 150 as well as the first image source 990A andthe second image source 992A. Occlusion may be also achieved byinstructing the display 992A not to emit light, rather than relying onan occlusion system 150. The occlusion system 150 may be replaced by areal-world occlusion object 155A shown in FIG. 11C.

In an embodiment, as illustrated in FIGS. 11D-E, a display system may becomprised of an optical combining system 101 which may include 1) afirst input interface configured to receive light along a first set oflight paths 131G or 132G from a first image source 990A, wherein thelight from the first image source 990A is operable to define a firstimage surface 991A; and 2) a second input interface configured toreceive light 133Y in FIG. 11D or 996A in FIG. 11E along a second set oflight paths from a second image source 123A in FIG. 11D or 992A in FIG.11E, wherein the light from the second image source is operable todefine a second image surface 123AS in FIG. 11D or 993A in FIG. 11E. Thedisplay system may also be configured to receive combined image light(e.g. 131G, 132G, and 133A in FIGS. 11D and 131G, 132G, and 996B in FIG.11E) from the optical combining system 101 and relay the received lightto relayed locations (e.g. 992A and 123B in FIG. 11D, and 992A and 994Ain FIG. 11E), whereby first and second relayed image surfaces (e.g.images on 992A or the surface 123BS of the relayed image 123B of thereal-world object in FIG. 11D, or images on 992A and 994A in FIG. 11E)are observable at the respective relayed locations. The display systemmay also be comprised of an occlusion system optically preceding atleast one of the first and second input interface (occlusion regions 188on occlusion layers 151A, 151B, and 151C), the occlusion systemconfigured to occlude a portion of at least one of the first and secondimage surfaces (123AS in FIG. 11D, 993A in FIG. 11E), wherein theoccluded portion corresponds to a relayed occluded portion (189) of atleast one of the first and second relayed image surfaces (123BS in FIG.11D, or 994A in FIG. 11E), the relayed occluded portion being occludedby the other one of the first and second relayed image surfaces (123BSmay be occluded by images on surface 992A in FIG. 11D, and images onsurface 994A may be occluded by images on surface 992A in FIG. 11E).Alternatively, the occlusion system shown in FIG. 11C may be utilizedwherein the occlusion of at least one of the first and second relayedimage surfaces (123BS in FIG. 11D, or 994A in FIG. 11E) may be achievedwith a real-world occlusion object 155A disposed in front of the firstor second image surfaces. More generally, and as demonstrated in FIGS.11A-D, the at least one of the first and second image sources comprises:a 2D display surface, a stereoscopic display surface, anautostereoscopic display surface, a multi-view display surface includingthe display surface of a horizontal parallax-only or HPO display, thesurfaces within a volumetric 3D display, a light field display surface,the surface of a real-world object emitting light, or the surface of areal-world object reflecting light. In an embodiment, at least one ofthe first and second image surface comprises: an image surface projectedfrom a 2D display surface, an image surface projected from astereoscopic display surface, an image surface projected from anautostereoscopic display surface, an image surface projected from amulti-view display surface, an image surface of a volumetric 3D display,a surface of a holographic object formed by light paths projected from alight field display, a surface of a real-world object, or a relayedimage of the surface of the real-world object. The characteristics ofthe occlusion system, optical fold systems, and ambient light rejectionshown in FIGS. 11D-E has been described in reference to FIGS. 11A-Cabove.

It is possible that an optical system may contain a first inputinterface configured to receive light along a first set of light pathsfrom a first image source, wherein the light from the first image sourceis operable to define a first image surface, a second input interfaceconfigured to receive light along a second set of light paths from asecond image source comprising a light field display, and a relay systemconfigured to direct the received light from the first and second imagesources to a viewing volume defined by boundary 1060, wherein at leastone of the first and second image surfaces is relayed by the relaysystem into the viewing volume defined by boundary 1060. Light from onlyone of the first or second image sources may be relayed. FIGS. 8A-Cdemonstrate relay configurations with two sources, where the relayitself combines the light from the two sources. FIG. 11F illustrates anoptical display system wherein the relay 5002 accepts light paths fromtwo image sources and simultaneously combines and relays the lightpaths. The relay 5002 may be the relay 5090 shown in FIG. 9G, or therelay 5080 shown in FIG. 9A with an image combiner placed between thetwo relay elements 5030A and 5030B to accept light paths from a secondimage source (see FIG. 9J). In FIG. 11F, the relay 5002 has a firstinput interface configured to receive light along a first set of lightpaths 133A from a first image source 123A, wherein the light from thefirst image source is operable to define a first image surface 123AS onthe surface of a real-world object 123A which may take the form of anemissive surface 123AS or a reflective surface 123AS. A second interfaceof relay system 5002 is configured to receive a second set of lightpaths 131A and 132A from second image source light field display 1001Awhich are determined according to a four-dimensional function defined bythe light field display 1001A such that each projected light path 131Aand 132A has a set of spatial coordinates and angular coordinates in afirst four-dimensional coordinate system defined with respect to adisplay screen plane 1021A of the second image source. The light 131A,132A from the second image source is operable to define second imagesurfaces 121A and 122A comprising holographic image surfaces. The relaysystem 5002 is configured direct the received light 121A, 122A from thesecond image source 1001A and the received light 133A from first imagesource 123AS to a viewing volume defined by boundary 1060 near virtualplane 1022A, wherein at least one of the first 123A and second 121A/122Bimage surfaces and in this case both are relayed by the relay systeminto the viewing volume defined by boundary 1060. In FIG. 11F, the relaysystem 5002 relays the received light 131A, 132A forming image surfaces121A, 122A into light paths 131B, 132B forming relayed image surfaces121B, 122B, respectively. The relay system 5002 also relays the receivedlight 133A from real-world image surface 123AS into light rays 133Bforming relayed surface 123BS.

In FIG. 11F, a controller 190 may be connected to the occlusion system150 as well as the image source light field display 1001A and issuedisplay instructions to the light field display 1001A and simultaneouslyissue occlusion instructions to the one or more occlusion layers 151,152, and 153 in occlusion system 150 in order to correctly occlude therelayed surface of the real-world object 123BS behind one or more of therelayed holographic surfaces 121B and 122B as viewed by a viewer 1050anywhere in the viewing volume defined by boundary 1060 of the relayedobjects 121B, 122B, and 123B. In FIG. 11F, both the first 123A andsecond 121A/122A image surfaces are relayed by the relay system 5002into the viewing volume near observer 1050 to define first 123B andsecond 121B/122B relayed image surfaces, respectively, and wherein theoccluded portion 188 of the light 133A corresponds to a relayed occludedportion 189 of at least one of the first 123B and second 121B/122Brelayed image surfaces (in this case the first relayed image surface123B), the relayed occluded portion being observable in the viewingvolume defined by boundary 1060 near observer 1050 as being occluded bythe other one of the first and second relayed image surfaces (in thiscase 121B). In an embodiment, at least one occlusion layer may have oneor more individually addressable elements, which may be occlusion sitesconfigured to block a portion of incident light or parallax barriers.The occlusion layers with individually addressable occlusion elementsmay be one or more transparent LED panels, transparent OLED panels, LCpanels, or other panels operable to selectively occlude light or formparallax barriers. Alternatively, the occlusion system shown in FIG. 11Cmay be utilized wherein the occlusion of at least one of the first andsecond relayed image surfaces (123BS in FIG. 11F) may be achieved with areal-world occlusion object (155A in FIG. 11C) disposed in front of thefirst or second image surfaces (123A in FIG. 11F). In this case, thecontroller 190 may issue instructions to a motion controller whichchanges the position of the real-world occlusion object in coordinationwith the movement of a relayed holographic object 121B, as demonstratedin FIG. 11C. In an embodiment, a distance between the at least oneocclusion layer 152 and the background image source 123A issubstantially equal to a distance between a foreground relayed surface121B and the relayed background surface 123B. In another embodiment, theocclusion region 188 defined by the at least one occlusion layer 152 isrelayed to the viewing volume defined by boundary 1060 to substantiallycoincide with the foreground surface 121B. In an embodiment, acontroller 190 is operable to coordinate a movement of the occlusionregion 188 (or the position of a real-world occlusion object such as155A in FIG. 11C) with a movement of an image surface 121B or 122B inthe viewing volume defined by boundary 1060. In an embodiment, the firstimage source 123A comprises: a 2D display surface, a stereoscopicdisplay surface, an autostereoscopic display surface, a multi-viewdisplay surface, the surface or surfaces of a volumetric 3D display, asecond light field display surface, the surface of a real-world objectemitting light, or the surface of a real-world object reflecting light.In an embodiment of FIG. 11F, an additional occlusion system comprisedof a real-world occlusion object (e.g. 155A in FIG. 11C) or one or moreocclusion planes (e.g. 150) optically preceding the second inputinterface of the relay 5002 in the path of light rays 131A and 132A maybe configured to occlude a portion of light from the light field display1001A corresponding to a portion of relayed holographic surfaces 121B or122B which may be occluded by relayed first image surface 123B in theevent that 123B is relayed in front 121B or 122B. In an embodiment, thesize and shape of the at least one occlusion region 188 or occlusionobject (not shown, but similar to 155A in FIG. 11C) is scaled to that ofthe foreground surface 121B in the viewing volume defined by boundary1060 so that an occluded portion 189 of the background surface 123Bcannot be observed behind the foreground surface 121B. In an embodiment,light from the first 123A and second 1001A image sources are bothrelayed into the viewing volume defined by boundary 1060 to form firstrelayed image surface 123B and second relayed image surfaces 121B, 122B,respectively. The first and second relayed image surfaces may be bothobservable by 1050 in a foreground, both observable in a background, orone may be in the foreground and the other one in the background.

The relay 5002 of the display system shown in FIG. 11F may be the relay5090 shown in FIG. 9G comprised of two transmissive reflectors 5030placed on parallel planes and separated from one another with an imagecombiner 101F disposed between them. The first transmissive reflectorrelay subsystem offers a first input interface configured to receivelight from a first image source which is the surface of real-worldobject 123A and is operable to relay the received light to a define afirst relayed image surface of the real-world object 123A and bereceived by an image combiner, the first relayed image surface having adepth profile different from a depth profile of the respective imagesurface 123A. The relay system 5090 further comprises an image combiningelement positioned to combine light from the first relay subsystemforming the relayed surface of real-world object surface 123A and thelight from the second image source defining a holographic surface,wherein the combined light comprising the first relayed image surfaceand the holographic surface is directed to the second relay subsystem,which is configured to relay the combined light to the viewing volumedefined by boundary 1060 near viewer 1050. The image combiner offers afirst interface to receive light from the surface 123AS of the firstimage source 123A, and this light is combined with the light from thesecond image source 1001A and relayed to a viewing volume 1060 nearviewer 1050 by the second transmissive reflector relay subsystem. Thesurface of real-world object 123A is relayed twice to 123B, while thesurfaces of projected holographic objects 121A, 122A ae relayed once to121B, 122B, respectively. For this reason, the depth profile of the oncerelayed holographic surfaces 121B, 122B is reversed, while the depthprofile of the twice-relayed holographic surface 123B of real-worldobject 123A is not reversed. In other words, the relay system 5002comprises a second relay subsystem (e.g. 5030G in FIG. 9G) configured torelay the first relayed image surface relayed from surface 123AS torelay locations in the viewing volume 1060 near observer 1050 to definea second relayed image surface 123B corresponding to the respectiveimage surface 123A defined by light from the first image source 123A,the second relayed image surface 123B having a depth profile that issubstantially the same as depth profile of the respective image surface123A defined by light from the first image source 123A. In anembodiment, holographic surfaces 121A, 122A defined by light paths 131A,132A projected from the light field display 1001A have first projecteddepth profiles with respect to screen plane 1021A, respectively, and theholographic surfaces are relayed by the relay system to define firstrelayed image surfaces 121B, 122B comprising relayed holographicsurfaces with first relayed depth profiles relative to virtual plane1022A that are different from the corresponding first projected depthprofiles. In an embodiment, the light field display comprises acontroller 190 configured to receive instructions for accounting for thedifference between the first projected depth profiles and the firstrelayed depth profiles by operating the light field display 1001A tooutput projected light such that the first relayed depth profiles of thefirst relayed image surfaces are the depth profiles intended for aviewer. In another embodiment, relayed locations of the first relayedimage surfaces 121B, 122B are determined according to a second 4Dfunction defined by the relay system 5002, such that light from thelight field display 1001A is relayed along respective relayed lightpaths 131B, 132B each having a set of spatial coordinates and angularcoordinates in a second 4D coordinate system, and the light fielddisplay 1001A comprises a controller 190 configured to receiveinstructions for accounting for the second 4D function by operating thelight field display 1001A to output light according to the first 4Dfunction such that the positional coordinates and angular coordinates inthe second 4D coordinate system for the relayed light paths 131B, 132Ballow the relayed image surfaces 121B, 122B to be presented to a viewer1050 as intended. This is discussed in detail with reference to FIG. 5Dabove.

In an embodiment the optical display system of FIG. 11F may furthercomprise an optical fold system optically preceding one of the first andsecond interfaces of relay 5002. These optional optical fold systems arelabelled 1150, 1160, or 1170 located in the paths of light 133A fromfirst image source 123A or located in the light paths 131A and 132A fromsecond image source 1001A in FIG. 11F. Optical fold system 1150 isdescribed in detail above in reference to FIGS. 10A-B, while selectiveoptical fold systems 1160 and 1170 are described above in detail inreference to FIGS. 10C and 10D, respectively. In an embodiment, theoptical fold system 1150, 1160, or 1170 comprises a plurality ofinternal optical layers, and light from the respective image source isdirected along a plurality of internal passes between internal opticallayers thereby increasing an optical path distance between the relaysubsystem and image surface locations in the viewing volume defined byboundary 1060. In an embodiment, one image source comprises the lightfield display 1021A, and wherein the optical fold system is located inthe path of the light 131A and 132A from the light field display toincrease the optical path length distance between respective imagesurface locations in the viewing volume near observer 1050 and the relaysystem 5002. In an embodiment, one image source comprises the lightfield display 1001A, and wherein the optical fold system is located inthe path of the second image source 123A to increase the optical pathlength distance between respective image surface locations such as 123Bin the viewing volume defined by boundary 1060 near viewer 1050 and therelay system 5002. In another embodiment, the optical system shown inFIG. 11F may further comprise an optical fold system optically followingat least one of the first and second interfaces of the relay system,within the internal layers of the relay system 5002 or on the output ofthe relay system 5002 in the path of light rays 131B, 132B, and 133B. Inan embodiment, the optical system shown in FIG. 11F has an environmentallight rejection system as shown in FIG. 11C which comprises an enclosure(e.g. 1080 in FIG. 11C) that partially encloses the relay system and awindow comprising a polarization filter (e.g. 1081 in FIG. 11C). In afurther embodiment, the polarization filter is operable to block ambientlight having a first polarization state. The ambient light has may havea first polarization state and is provided by a light source comprisinga polarization output filter configured to allow light only of the firstpolarization state to pass through (e.g. light source 1085 beingfiltered by polarization output filter 1082 in FIG. 11C).

The relay 5002 of the display system shown in FIG. 11F relays firstemissive or reflective surface 123AS from first image source real-worldobject 123A as well as second holographic image surfaces 121A, 122Aprojected by second image source light field display 1001A. In anembodiment, the optical system shown in FIG. 11F may be comprised of arelay which receives sets of light paths from these two image sourcesand directs this light to a viewing volume defined by boundary 1060, butwherein only one set of light paths from one of the image sources isrelayed. FIG. 11G is the display system in FIG. 11F wherein the relay5002 which relays image surfaces from two sources has been replaced byrelay 5003 which only relays the image surfaces projected from onesource, the light field display 1001A, while directly passing light fromthe other image source real-world object 123A to the viewing volume nearobserver 1050. The numbering of FIG. 11F is used in FIG. 11G. The relay5003 may be the relay system 5020 shown in FIG. 1B with only oneretroreflector 1006B, the relay system 5050 shown in FIG. 5E with onlyone reflective mirror 1007B, relay system 5060 shown in FIG. 5F withonly one reflective Fresnel mirror 1008B, or some other relay whichsimultaneously relays light from a first interface while directlypassing light that arrives from a second interface. Each of these relays5020, 5040, and 5050 may be comprised of a beam splitter and a focusingelement (e.g. a retroreflector for 5020 or a reflective focusing mirrorfor 5040 and 5050) disposed opposite to a first relay interface whichaccepts light from the light field display 1001A. Projected holographicsurfaces 121A and 122A will be relayed by the first interface of theserelay configurations 5020, 5040, and 5050, while light from thereal-world object 123A received on the second relay interface will passdirectly through the beam splitter of the relay and to observer 1050without being actively relayed.

An observer 1050 in a viewing volume defined by boundary 1060 may seetwo foreground relayed holographic surfaces 121B and 122B in front of areal-world background object 123A which produces light 133A which passesdirectly through the relay 5003. An occlusion system 150 comprised ofocclusion planes, or a real-world occlusion object like 155A shown inFIG. 11C may be used to occlude the portion of the real-world backgroundobject 123A behind one or more relayed holographic surfaces 121B and122B. In an embodiment, only one of the first and second image surfaces(e.g. 121A/122A, but not 123AS in FIG. 11G) is relayed into the viewingvolume near viewer 1050 to define a relayed image surface 121B/122B inthe viewing volume defined by boundary 1060, and wherein the occludedportion of the light (e.g. 133A in FIG. 11G) corresponds to an occludedportion of the other one of the first and second image surfaces (e.g.123AS) observable in the viewing volume as being occluded by the relayedimage surface (e.g. 121B/122B).

In an embodiment, the light field display 1001A in FIGS. 11F and 11Ginstead may be another type of display. FIGS. 11H, 11I, and 11J beloware embodiments of an optical system comprising a first input interfaceconfigured to receive light 133A along a first set of light paths from afirst image source 123A, wherein the light from the first image sourceis operable to define a first image surface 123AS; a second inputinterface configured to receive light along a second set of light pathsfrom a second image source, wherein the light from the second imagesource is operable to define a second image surface; a relay systemconfigured to direct the received light from the first and second imagesources to a viewing volume defined by boundary 1060, wherein at leastone of the first 123A and second image surfaces is relayed by a relaysystem 5002 or 5003 into the viewing volume near viewer 1050; and anocclusion system 150 or 155A configured to occlude a portion of lightfrom at least one of the first and second image sources. FIG. 11H is thedisplay system of FIG. 11F with the second image source light fielddisplay 1001A replaced by second image source display 990A with displaysurface 991A. In an embodiment, the second image source may be the a 2Ddisplay surface, a stereoscopic display surface, an autostereoscopicdisplay surface, a multi-view display surface which may be the surfaceof a horizontal parallax-only HPO multi-view display such as alenticular display, the surface or surfaces of a volumetric 3D display,the surface of a real-world object emitting light, or the surface of areal-world object reflecting light. Some of the numbering of FIG. 11F isused in FIG. 11H. Light rays 131G and 132G from the second image sourcedisplay 990A with surface 991A are relayed to light paths 131H and 132H,respectively, and are focused on relayed virtual display plane 992A.Real-world object 123B is relayed to the same place as shown in FIG.11F. Occlusion planes 151-153 may be activated to block out some of thelight from real-world object 123A, so that portions of the relayed imageof the real-world object cannot be seen behind images that are relayedto the relayed virtual display plane 992A. The controller 191 may beconnected to the occlusion system 150 as well as the first image sourcedisplay 990A and possibly optional selective optical folding systems1160 or 1170 if they are in place. In an embodiment, the first imagesource real-world object 123A as well as the second image source display990A may be replaced by any of: a 2D display surface, a stereoscopicdisplay surface, an autostereoscopic display surface, a multi-viewdisplay surface which may be the surface of a horizontal parallax-onlyHPO multi-view display such as a lenticular display, the surface orsurfaces of a volumetric 3D display, the surface of a light fielddisplay, the surface of a real-world object emitting light, or thesurface of a real-world object reflecting light. The first image surface123AS as well as the second image surface 991A may be any of: an imagesurface projected from a 2D display surface, an image surface projectedfrom a stereoscopic display surface, an image surface projected from anautostereoscopic display surface, an image surface projected from amulti-view display surface, the image surface of a volumetric 3Ddisplay, the surface of a holographic object, a surface of a real-worldobject, or a relayed image of the surface of the real-world object.

FIG. 11I is the display system of FIG. 11F with the second image sourcelight field display 1001A replaced by second image source real-worldobject 998A, and an occlusion system comprised of real-world occlusionobject 155A used in place of the occlusion system 150 having one or moreocclusion planes 151, 152, and 153. Light rays 131K and 132K from thereal-world object 998A are received by the relay and relayed to lightpaths 131H and 132H, respectively, forming relayed object 998B withrelayed surface 998BS. A real-world occlusion object 155A may be placedto occlude a portion of the light 133A from the first image sourcereal-world object 123A. In an embodiment both the first 123AS and second998AS image surfaces are relayed by the relay system 5002 into theviewing volume defined by boundary 1060 to define first 123BS and second998BS relayed image surfaces, respectively, and wherein the occludedportion of the light corresponds to a relayed occluded portion of atleast one of the first and second relayed image surfaces, in thisexample first image surface 123AS, the relayed occluded portion 189being observable in the viewing volume near viewer 1050 as beingoccluded by the other one of the first and second relayed imagesurfaces, in this example second relayed image surface 999BS which willappear to block out a portion 189 of the light rays from backgroundrelayed image surface 123BS to observer 1050 when foreground relayedreal-world object surface 999BS is in front of background relayedreal-world object 123B. A controller 191 may be connected to a motioncontroller imparting motion 156A to the occlusion object 155A. In anembodiment, real-world objects 998A or 123A may be on a motorized stagecontrolled by controller 191, and the controller 191 may simultaneouslyadjust the position of the real-world object and change the location ofthe occlusion object 155A in order to keep the background relayedsurface 123BS occluded when it is behind the foreground relayed surface998BS.

FIG. 11J is the display system of FIG. 11I with the relay 5002 replacedby relay 5003. The relay 5003 may be the relay system 5020 shown in FIG.1B with only one retroreflector 1006B, relay system 5050 shown in FIG.5E with only one reflective mirror 1007B, relay system 5060 shown inFIG. 5F with only one reflective Fresnel mirror 1008B, or some otherrelay which simultaneously relays light from a first interface whiledirectly passing through light that arrives from a second interface.Each of these relays 5020, 5040, and 5050 may be comprised of a beamsplitter and a focusing element (e.g. a retroreflector for 5020 or areflective focusing mirror for 5040 and 5050) disposed opposite to afirst relay interface which accepts light from a second image source998A which defines image surface 998AS. In an embodiment, only one ofthe first 123AS and second 998AS image surfaces, here the second imagesurface 998AS, is relayed into the viewing volume defined by boundary1060 near observer 1050 to define a relayed image surface 998B in theviewing volume, and wherein the occluded portion of the light 133Acorresponds to an occluded portion of the other one of the first andsecond image surfaces which is not relayed, here first image source 123Aobservable in the viewing volume defined by boundary 1060 as beingoccluded by the relayed image surface 998B.

FIG. 12 shows a display system 1200 comprised of the display systemshown in FIG. 11A, where the relay system 5000 is realized by atransmissive reflector 5030, and there are no optical fold systems 1150,1160, or 1170 illustrated. The numbering of FIG. 11A is used in FIG. 12. Relayed holographic object surfaces 121B/122B are located at relayedlocations distributed around a virtual display plane 1022A, and therelayed image surface 123B of the real-world object 123A is projectedclose to the relayed holographic objects 121B and 122B.

FIG. 13 shows the display configuration shown in FIG. 12 , except thatan optical fold system 1150 has been placed between the light fielddisplay 1001A and the beam splitter 101 of the optical combining system.The numbering of FIG. 12 is used in FIG. 13 . FIG. 13 is the displaysystem shown in FIG. 11A with the relay system comprised of atransmissive reflector relay 5030. The effective optical path length ofthe optical fold system 1150 is about three times the distance D 1151,where D 1151 is the length of Path 2 or Path 3 shown in FIG. 10B. Theresult is that the diverging rays 131A forming the holographic objectsurface 121A have enough optical path length to spread out into rays131B, which are relayed into rays 131C which will converge at a furtherdistance from the transmissive reflector 5030 than the convergencedistance with no optical fold system 1150. Similarly, the diverging rays132A forming holographic object 122A spread out into rays 132B as aresult of the optical fold system 1150, which are relayed to light rays132C. In FIG. 13 , holographic object surfaces 121X and 122X at relayedlocations around virtual display plane 1022X show the location of therelayed holographic object surfaces 121B and 122B shown in FIG. 12 withno optical fold system 1150, respectively, while holographic objectsurfaces 121B and 122B at relayed locations around virtual display plane1022A show the location of the relayed holographic object surfaces withthe optical fold system 1150 present. The offset 1152 between virtualdisplay plane 1022X and 1022A is 2D, where D is the effective pathlength 1151 of the optical fold system 1150 placed in the path of thelight field display 1001A. In another embodiment, the optical foldsystem 1150 is placed in the path of the real-world object 123A, whichacts to move just the relayed real-world image surface 123B closer tothe observer 1050. In a different embodiment, the optical fold system1150 may be placed between the beam splitter 101 and the relay system5030, acting to move both the relayed holographic objects and therelayed real-world image closer to the observer. In still anotherembodiment, the optical fold system 1150 may be placed between the relaysystem 5030 and the relayed real-world image surface 123B, resulting inthis relayed image 123B as well as the holographic object surfaces 121Band 122B moving closer to the observer 1050. Note the reversal of depthshown in FIG. 13 . The depth ordering of the relayed holographic objects121B and 122B around virtual display screen 1022A is reversed from thedepth ordering of directly projected object surfaces 121A and 122Arelative to the display screen plane 1021A, respectively. Similarly, therelayed image surface 123B of the real-world object 123A is also depthreversed as shown by how the curved face of the real-world object 123Ais relayed. Under the circumstance in which the real-world object 123Ais complex, such as a real person's face or a complex real-worldbackground scene, and cannot be easily built with depth reversal, it ispossible to replace the real-world object 123A by the relayed surface ofa real-world object with reversed depth. In an embodiment, the opticalfold system 1150 may be replaced with a selective optical fold system1160 or 1170 described above. In this embodiment, only one group oflight rays 131B or 132B may have their optical path length extended,resulting in only one of the relayed objects 121B or 122B being relayedcloser to observer 1050.

FIG. 14A shows a display system 1400 which is modified from the displaysystem configuration shown in FIG. 13 by an extra relay for thereal-world object 123A. In FIG. 14A, an input relay system 5030A is usedto relay the image surface 123A of the real-world object to form anintermediate, depth-reversed, relayed image 123B of the real-worldobject, which is then received by relay system 5030 and relayed onceagain with depth reversal to form a depth-correct relayed real-worldimage surface 123C. FIG. 13 is the display system shown in FIG. 11A withthe relay system comprised of a transmissive reflector relay 5030, andwherein the surface of real-world object 123A is relayed twice. Notethat the only difference between real-world image surface 123A and therelayed real-world image surface 123C is that the image is up-downflipped, a feature that may be corrected with a 180 degree rotation ofthe position of real-world object 123A. The capability of the relaysystem comprised of relays 5030 and 5030A in display system 1400 shownin FIG. 14A to relay images of real-world objects without depth reversalallows images of complex real-world dynamic objects to be relayedreal-time so they may be displayed alongside relayed holographic objectsurfaces 121B and 122B relayed from the light-field display 1001A. Inthis configuration, the angular light field coordinates u and v may bereversed computationally for the holographic object surfaces 121A and122A projected by the light field display 1001A in order to achieve thecorrect depth profile desired for relayed holographic image surfaces121B and 122B, as discussed above in regard to FIGS. 1A and 1B. In FIG.14A, the occlusion system 150 could be replaced by a real-worldocclusion object like object 155A in FIG. 11C. Also, as shown in FIGS.11D-E above, the first image source light field display 1001A surface1021A and the second image source real-world object 123A surface mayeach be replaced by any of: a 2D display surface, a stereoscopic displaysurface, an autostereoscopic display surface, a multi-view displaysurface which may be the surface of a horizontal parallax-onlymulti-view display such as a lenticular display, the surface or surfacesof a volumetric 3D display, the surface of a real-world object emittinglight, or the surface of a real-world object reflecting light.

FIG. 14B shows a display system 1410 which is modified from the displaysystem configuration shown in FIG. 12 by an extra relay for thereal-world object 123A. The numbering of FIG. 12 is used in FIG. 14B.FIG. 14B is the display system shown in FIG. 11F with the relay systemcomprised of a transmissive reflector relay 5030, and wherein thesurface of real-world object 123A is relayed twice. In FIG. 14B, aninput relay 5030A is used to relay the light rays 133K from the surfaceof a real-world object 123A to once-relayed light rays 133L which forman intermediate, depth-reversed, relayed surface 123B of the real-worldobject 123A. A first portion of the light rays 133L which form therelayed surface 123B reflect from the surface of the transmissivereflector 5030 into light rays 133LR observable by viewer 1050, while asecond portion of the light rays 133L are relayed by relay 5030 intolight rays 133M which form the twice-relayed surface 123C of real-worldobject surface 123A. The fraction of once-relayed light 133L which isreflected into light paths 133LR toward the observer 1050 may be tunedby selecting the reflectivity of the surface of relay 5030. While thetwice-relayed surface 123C of real-world object 123A is relayed to aposition opposite of relay 5030 from the viewer 1050, the reflectedlight rays 133LR reaching viewer 1050 substantially line up with lightrays 133M forming the surface 123C and are thus observed by viewer tooriginate from twice-relayed surface 123C of real-world object 123A.Observer 1050 sees the relayed holographic object surfaces 121B and 122Bas well as the back of surface 123C. On the opposing side of the relay5030, an observer 1050A will see the back of relayed holographic object121B by receiving a reflected portion 131AR of the incident light rays131A forming holographic object 121A, the back of relayed holographicobject 122B by receiving a reflected portion 132AR of the incident lightrays 132A forming holographic object 122A, and the front oftwice-relayed surface 123C of real-world object surface 123A formed bylight rays 133M. In this configuration, the angular light fieldcoordinates u and v may be reversed computationally for the holographicobject surfaces 121A and 122A projected by the light field display 1001Ain order to achieve the correct depth profile desired for relayedholographic image surfaces 121B and 122B, as discussed above in regardto FIGS. 1A and 1B. In FIG. 14B, the occlusion system 150 could bereplaced by a real-world occlusion object like object 155A in FIG. 11C.Also, as shown in FIGS. 11D-E above, the first image source light fielddisplay 1001A surface 1021A and the second image source real-worldobject 123A surface may each be replaced by any of: a 2D displaysurface, a stereoscopic display surface, an autostereoscopic displaysurface, a multi-view display surface which may be the surface of ahorizontal parallax-only multi-view display such as a lenticulardisplay, the surface or surfaces of a volumetric 3D display, the surfaceof a real-world object emitting light, or the surface of a real-worldobject reflecting light.

FIG. 15 is the display system configuration shown in FIG. 11A, with therelay 5020 used with an optical folding system 1150 in the path of thelight 131A and 132A from the light field display 1001A. Theconfiguration of FIG. 15 is similar to the configuration shown in FIG.13 , except that instead of a relay system comprised of a transmissivereflector 5030, the relay system 5020 is comprised of a beam splitter101B and one or more retroreflectors 1006A, 1006B, similar to theconfiguration 5020 shown in FIG. 1B. The numbering in FIG. 13 applies toFIG. 15 for similar elements, and some of the discussion of FIG. 1Bapplies to this relay configuration. In an embodiment in which anoptional additional retroreflector 1006B is included in the relay system5020, the additional retroreflector 1006B may be placed orthogonally tothe first retroreflector 1006A, and in some embodiments, the additionalretroreflector 1006B may be positioned at equal distance away from thebeam splitter 101B as the distance between the first retroreflector1006A and the beam splitter 101B. It is to be appreciated that theconfiguration of the relay system 5020 shown in FIG. 15 may beimplemented with: 1) only the retroreflector 1006A; 2) only theretroreflector 1006B; or 3) both retroreflectors 1006A and 1006Bincluded and aligned. In an embodiment, light rays 131A formingholographic object surface 121A and light rays 132A forming holographicobject surface 122A may have their optical path lengths extended withinthe optical fold system 1150, and become light rays 131B and 132B,respectively. In an embodiment, the light rays 131B and 132B from theholographic object surfaces 121A and 122A are received through a firstinput interface of an optical combining system 101A, and light 133Y froma second image source 123A is received through a second input interfaceof the optical combining system 101A. In an embodiment, the second imagesource comprises a real-world object 123A emitting or reflecting light.In an embodiment, a portion of the light 133Y from the real-world object123A is reflected from a beam splitter 101A of the optical combiningsystem into light rays 133A and is combined by the beam splitter 101Awith the light 131B and 132B from the holographic object surfaces 121Aand 122A. This combined image light 131B, 132B, and 133A is received bythe relay system 5020. In an embodiment, the retroreflector 1006A andthe beam splitter 101B of the relay system 5020 are aligned such thatthe combined light is directed from the beam splitter 101B in anapproach direction towards the retroreflector 1006A and is reflectedfrom the retroreflector 1006A along a return direction opposite of theapproach direction. Light along the return direction is directed towardsthe relayed locations around the relayed virtual screen plane 1022A. Inan embodiment, the retroreflector 1006A and the beam splitter 101B ofthe relay system 5020 are aligned such that a first portion of thecombined light 131B, 132B, and 133A is reflected by the beam splitter101B of the relay system 5020 toward the retroreflector 1006A. Uponreflecting from the reflector 1006A, the light paths are reversed, and aportion of these reversed paths pass through the beam splitter 101Balong light rays 131C, 132C, and 133B, being focused by the relay system5020 at relayed locations to form holographic object surfaces 121B,122B, and relayed surface 123B of the real-world object 123A,respectively. A second portion of the combined light 131B, 132B, and133A is received by relay system 5020 and is transmitted through thebeam splitter 101B toward the optional additional retroreflector 1006Balong an additional approach direction. These light paths reflect fromthe optional additional retroreflector 1006B along an additional returndirection opposite the additional approach direction towards the beamsplitter 101B, upon which they are reflected along substantially thesame light paths 131C, 132C, and 133B as the first portion of thecombined light from first retroreflector 1006A, contributing to formingholographic object surfaces 121B, 122B, and relayed surface 123B ofreal-world object 123A, respectively.

In the event that unpolarized light is received by the relay system5020, the addition of the optional additional retroreflector 1006B mayresult in increased brightness of the relayed holographic objectsurfaces 121B and 122B as well as relayed image surface 123B of thesecond image source 123A. A polarization beam splitter 101B may be usedto direct a first linear polarization of combined light 131B, 132B, and133A toward retroreflector 1006A, and a second linear polarization ofcombined light 131B, 132B, and 133A toward retroreflector 1006B. Thefirst linear polarization of light may be converted to a first circularpolarization by a quarter wave retarder 1041A before reflection by theretroreflector 1006A, which acts to change the reflected light to asecond circular polarization orthogonal to the first circularpolarization. Upon passing back through the quarter wave retarder 1041Atoward the beam splitter 101B, the reflected light is converted to asecond linear polarization orthogonal to the first. This state ofpolarization will pass through the beam splitter 101B withoutsignificant reflection. Similarly, the second state of linearpolarization of light directed at the optional retroreflector 1006B willbe converted into the orthogonal state of first linear polarization bypassing through the quarter wave retarder 1041B, reflecting from theoptional retroreflector 1006B, and passing through the quarter waveretarder 1041B a second time, and this first state of linearpolarization should be substantially reflected by the polarization beamsplitter 101B and contribute to imaging the relayed holographic imagesurfaces 121B and 122B, and the relayed image 123B of the real-worldobject. If the light received by the relay system 5020 is polarized,then a polarization beam splitter 101B may be used, and good performancemay be achieved with just the first retroreflector 1006A alone, withoutthe optional retroreflector 1006B. In other embodiments, the optionaloptical elements 1041A and 1041B may be polarization controllingelements apart from quarter wave retarders, refractive elements,diffractive elements, or other optical elements.

A technical advantage allowed by the relay configuration shown in FIG.15 is that relayed holographic object surfaces and a relayed imagesurface of a second image source, such as images of real-world objects,may be combined in substantially the same space, close to the relayedvirtual screen plane 1022A if desired. However, in some applications, itmay be desirable to relay the holographic object surfaces to aforeground in front of a background such as a real-world background.FIG. 16 is the display configuration of FIG. 11G comprising relay system5020 which simultaneously relays the surface of holographic objects andpasses light directly from a real-world background source through to anobserver. The relay 5020 in FIG. 16 comprised of a beam splitter and aretroreflector, in which holographic object surfaces 121A and 122Aprojected around a display plane 1021A are relayed to holographic objectsurfaces 121B and 122B around a virtual screen plane 1022A,respectively. In an embodiment, the relay system 5020 may be consideredas an optical combiner for the light from the real-world backgroundobject 123A and the holographic object surfaces 121A and 122A. FIG. 16shows a configuration for a relay system in which is similar to theconfiguration of FIG. 15 , except that the relay system 5020 containsonly a single retroreflector disposed on the opposite side of the beamsplitter 101 from the light field display 1001A, and the beam splitter101 also allows light 133A from the real-world object 123A to reach theobserver 1050 with a single pass through the beam splitter 101. Thenumbering of FIG. 15 is used in FIG. 16 for similar elements, and thedescription of the operation of the relay 5020 given for FIG. 15 withonly one retroreflector applies here. In an embodiment, an occlusionsystem may include one or more occlusion layers 151, 152, and 153 withindividually-addressable occlusion elements 188, and the occlusionlayers may be transparent, semi-transparent, or fully occluding. In FIG.16 , the observer 1050 views the relayed holographic object surface121B, but the pattern of occlusion elements 188 has been configured sothat the observer 1050 does not receive light from the portion of thereal-world background image surface 123A behind the holographic object121B, along the lines 132D illustrated as extensions of the rays 131B,so that the relayed holographic object surface 121B appears to occludethe real-world background image surface 123A in the same way that a realobject placed at relayed holographic object surface 121B would occludethe background image surface 123A. In an embodiment, a real-worldocclusion object like object 155A in FIG. 11C could replace theocclusion system comprised of occlusion layers 151, 152, and 153. Inanother embodiment, optional optical folding system 1150 shown in FIGS.10A-B, selective folding system 1160 shown in FIG. 10C, or selectivefolding system 1170 shown in FIG. 10D may be used in the light paths131B, 132B of relayed objects 121B, 122B, respectively. If selectiveoptical folding systems 1160 or 1170 are configured to only increase thepath lengths on light paths 131B and 132B, and not light paths 133A, andthe optical path length of these selective folding systems 1160 or 1170were made to be sufficiently long, then the observer 1050 may perceiverelayed holographic surfaces 121B and 122B to be behind the surface ofreal-world object 123A. In this instance, an occlusion system in thepath of the relayed image source light field display 1001A may provideocclusion of a background relayed object 121B or 122B behind thenon-relayed image surface 123A.

In an embodiment, it is possible to use relays with mirrored surfaces,which may include curved mirrors or Fresnel mirrors, to relayholographic object surfaces and image surfaces of real-world objects.FIG. 17 is display system with a relay configuration that is similar tothe relay configuration shown in FIG. 15 , wherein the relay system 5020comprised of retroreflector 1006A and optional additional retroreflector1006B has been replaced with relay system 5050 comprised of a mirroredsurface 1007A which may include a curved reflective mirror and anoptional additional mirrored surface 1007B, which may be orthogonallyplaced and may include a curved reflective mirror. Relay system 5050 isshown in FIG. 5E and is described above. FIG. 17 is the relay system ofFIG. 11B with the relay 5050 used in place of 5001. In FIG. 17 , ratherthan having an optical fold system 1150 placed in the light paths 131Aand 132A of the projected holographic object surfaces 121A and 122A,respectively, the optical fold system 1150 is placed in the light path133Y of the second image source, which may be a real-world object 123Aemitting or reflecting light. The magnification or minification of eachrelayed object surface may depend on the source object's distance to theeffective focal point of the mirror system, as described above inreference to the curved mirror relay configurations shown in FIGS. 4D,5D and 5E. In FIG. 17 , the light 133Y from a real-world object 123Apasses through an optical fold system 1150, into light rays 133A, inwhich the optical fold system 1150, as shown in FIGS. 10A and 10B,causes the relayed real-world image surface 123B to move further fromthe relay system 5050. The light 133A from the surface of real-worldobject 123A is received by a first input interface of beam splitter 101Aof the optical combining system, and light 131A and 132A fromholographic object surfaces 121A and 122A is received through a secondinput interface of the beam splitter 101A. The combined light isreceived the relay system 5050. The relay system 5050 and the detailedreflection of light within 5050 is described above with reference toFIG. 5E. A first fraction of received light 131A, 132A, and 133B isreflected from the beam splitter 101B to the right, next reflecting fromthe first mirror 1007A in a return direction opposite the approachdirection, and passes through the beam splitter 101B into light paths131C, 132C, and 133C, forming relayed image surfaces 121B, 122B, and123B, respectively. A second fraction of received light 131A, 132A, and133B is transmitted by the beam splitter 101B, and continues verticallyin an additional approach direction, reflecting from the optional mirror1007B in an additional return direction generally opposite theadditional approach direction, and next reflecting from the beamsplitter 101B into substantially the same light paths 131C, 132C, and133C, also contributing light to form relayed image surfaces 121B, 122B,and 123B, respectively. In an embodiment in which both mirrored surfaces1007A and 1007B are present, it may be desirable to match themgeometrically, be placed equal distance from the beam splitter 101B ofthe relay system 5050 and be orthogonal to one another. The relay system5050 may also be implemented with only one of the mirrored surfaces1007A or 1007B present. In an embodiment a linear polarization beamsplitter 101B is used, and the optional optical elements 1041A and 1041Bcomprising quarter wave retarders may be included to allow lightreturning to the beam splitter 101B after being reflected from amirrored surface 1007A or 1007B to be in a state of linear polarizationopposite to the state of linear polarization of the light approachingmirrors 1007A or 1007B, and this allows for reducing the unwantedreflections from beam splitter 101B as described above in reference toFIGS. 5C and 5E. The full light paths for rays 132A from holographicobject 122A and relayed rays 132C for the relayed holographic object122B are not shown in FIG. 17 for simplification (see the discussion ofFIG. 5E). Finally, an occlusion system, which may comprise individuallyaddressable occlusion regions 188 on the occlusion layers 151, 152, and153, may block relayed light from a portion of the surface of real-worldobject 123A, resulting in the observer 1050 not being able to see theblacked-out region 189 of the relayed image surface 123B of thereal-world object 123A behind the relayed holographic image surface122B, resulting in natural occlusion handling for the relayed backgroundimage surface 123B behind relayed holographic image surface 122B.

FIG. 18 is a display system which behaves like the display system ofFIG. 17 , but with a relay 5060 comprised of reflective Fresnel mirror1008A and optional reflective Fresnel mirror 1008B used in place of therelay system 5050 in FIG. 17 . The numbering from FIG. 17 is used inFIG. 18 for similar elements. FIG. 18 is the relay system of FIG. 11Bwith the relay 5060 used in place of 5001. As found in the abovediscussion of the relay system 5050 shown in FIG. 5E, the relay system5060 may be implemented with either Fresnel reflector 1008A or 1008Bremoved. The detailed reflections within the relay system 5060 aredescribed above for the discussion of 5060 in FIG. 5F.

FIG. 19 is the display system of FIG. 11G with a relay 5060 comprised ofan image combiner 101 and a Fresnel mirror 1008B, wherein the surface ofholographic objects are relayed by the relay 5060, and a real-worldbackground is visible through the relay 5060. The function of thedisplay system of FIG. 19 would be the same if relay 5060 were replacedby a relay 5050 by exchanging Fresnel mirror 1008B with a curved mirror1007B as shown in FIG. 5E. Holographic object surfaces 121A and 122Aaround a display plane 1021A are relayed to relayed holographic imagesurfaces 121B and 122B around a virtual screen plane 1022A,respectively. The relay system 5060 may be considered as functioning asan optical combiner for the light rays 131A and 132A from holographicobject surfaces 121A and 122A projected by the light field display1001A, respectively, and light rays 133A from the surface of real-worldbackground object 123A which merely pass through the optical combiner101. A portion of light rays 131A and 132A from the surfaces ofholographic objects 121A and 122A are received by the relay 5060,passing through the image combiner 101, reflecting from the Fresnelmirror 1008B into light rays 131B and 132B, and then reflecting from theimage combiner 101 toward light rays 131C and 132C, which converge toform the holographic objects 121B and 122B, respectively. The opticalfold system 1150, 1160, or 1170 described above is optional. In theexample shown in FIG. 19 , the observer 1050 viewing relayed holographicimage surface 122B may not be able to see the background real-worldobject surface 123A behind the relayed holographic image surface 122Bbecause of the operation of an occlusion system 150 with one or moreocclusion layers 151, 152, and 153, which as discussed above may includeindividually-addressable occlusion regions 188. The operation of theocclusion system 150 allows the observer 1050 to view the relayedholographic image surface 122B as it were a real object that occludesthe relayed background object surface 123B. Lines 132D are illustratedextensions of the light rays 132C forming relayed holographic imagesurface 122B, showing how an occlusion region 188 intersects each ofthese lines to attenuate or block these light rays. The occlusionpattern 188 may be determined experimentally, computationally,algorithmically, or using some other method.

Most of the relay systems shown above in this disclosure allow for relaylocations distributed about a relayed virtual screen plane, which isrotated at 90 degrees or 180 degrees from the light field display screenplane. FIG. 20 shows an example of a display system with an in-linerelay system 5100 comprised of a transmissive retroreflector 2051, areflective surface 2060, and several optical layers 2061, 2062, and 2063wherein the light field display screen plane 1021A and the relayedvirtual screen plane 1022A are parallel. Some of the optical layers2061, 2062, and 2063 are optional. The reflector 2060 of the relaysystem 5100 is configured to receive the rays 2071 projected from thelight field display 1001A and reflect the received light into rays 2072,and the retroreflector 2051 is configured to retroreflect these lightrays 2072 into light rays 2073 which trace the reverse path beforeleaving the relay system 5100. The transmissive retroreflector 2051 actsto focus the rays 2073, creating a relayed virtual screen plane 1022A.There are a number of configuration options for the optical layerswithin relay system 5100. In one embodiment, the reflector 2060 mayinclude a half mirror, while in other embodiments the reflector 2060 mayinclude a reflective polarizer. In the case where reflector 2060 is areflective polarizer, the reflector 2060 may reflect light of a firststate of linear polarization L1, and transmit the orthogonal secondstate of linear polarization L2, or the reflector 2060 may be configuredto reflect a first state of circular polarization C1, and transmit asecond state of circular polarization C2. If the reflector 2060 is areflective polarizer, then the optical layers 2061, 2062, and 2063 maybe configured to set the polarization of the light 2071 firstapproaching the reflective polarizer 2060 to a first state which will bereflected by the rays 2071, and set the state of the light 2073approaching the reflective polarizer 2060 on the second pass to a secondstate of polarization orthogonal to the first state so it will passthrough the reflective polarizer 2060. This can be achieved severalways. In an example, if the reflective polarizer 2060 reflects a firststate of linear polarization L1, and transmits a second state of linearpolarization L2, orthogonal to the first state L1, then the lightapproaching the reflector 2060 on light rays 2071 should be of linearpolarization L1, and the light approaching the reflector 2060 on lightrays 2073 should be of linear polarization state L2. To achieve this,optical layer 2061 can be configured to include a polarizing filter,which absorbs state L2 and transmits state L1. Alternatively, in anembodiment in which the display produces light only in the L1 state,like some LC panels, the layer 2061 may be omitted. Optical layer 2062can be a quarter wave retarder with a fast axis angle of 45 degrees, andoptical layer 2063 on the opposite side of the retroreflector 2051 maybe a quarter wave retarder with the opposite fast axis angle of −45degrees. In this configuration, light rays 2071 may have both L1 and L2states of polarization at point A, contain only the L1 state ofpolarization at point B, be converted into a first state of circularpolarization C1 at point C, which will pass through the retroreflectorto point D, and be converted back into the L1 state of polarization atpoint E, reflect into light rays 2072 at point F as the L1 state, becomethe first state of circular polarization C1 at point G, reflect intolight rays 2073 with the reverse second state of circular polarizationC2 at point H as a result of the reflection, be converted into thesecond state L2 of linear polarization at point I, passing through thetransmissive reflector 2060 at point J. In other embodiments, thereflector 2060 may be a reflective polarizer, which transmits a firststate of circular polarization C1, and reflects a second orthogonalstate of circular polarization C2, with or without a change of C2 to C1for the reflected light. In addition, it is possible that thetransmissive retroreflector 2051 is configured to be polarizationdependent, so that it transmits a first state of polarization, andreflects or absorbs a second state of polarization, orthogonal to thefirst, with these states of polarization linear ones L1 and L2 orcircular ones C1 and C2.

The relay system 5100 including the transmissive retroreflector 2051described above will reverse the depth profiles of object image surfacesand the corresponding relayed image surfaces. FIG. 21A shows holographicobject surfaces 121Z and 122Z projected from a LF display 1001A andviewed by an observer 1048. For these holographic objects to be relayedby the relay system 5100 so they appear in the same orientation relativeto a virtual screen plane as they are relative to the display screenplane 1021A, the u-v angular coordinates may have their polaritiesreversed as shown in FIGS. 2B and 2C. FIG. 21B shows the projection ofholographic object surfaces 121A and 122A obtained when all the u-vangular coordinates in FIG. 21A have been reversed. FIG. 21C is a viewof a display system demonstrating how the holographic objects shown inFIG. 21B may be relayed by utilizing a relay system 5100 including atransmissive retroreflector 2051 shown in FIG. 20 . Light rays 131A and132A which form holographic object surfaces 121A and 122A, respectively,pass through the transmissive retroreflector 2051 as well as opticallayers 2061, 2062, and 2063 in a first approach pass as they diverge inadvance of reflecting from the reflector 2060. The reflected rays 131Band 132B, in a first return pass, continue to diverge as they passthrough one optical layer 2063 before being retroreflected fromtransmissive reflector 2051 in a second approach pass, forming lightrays 131C and 132C, which are now focused to form relayed holographicimage surfaces 121B and 122B, respectively. LF display screen plane1021A is relayed to virtual screen plane 1022A. Observer 1050 in FIG.21C sees the same distribution of holographic objects as observer 1048in FIG. 21A, and the same depth profile of these holographic objects.

FIG. 22 shows a display system which uses a relay system 5100 with atransmissive retroreflector 2051, employs an optical fold system 1150,and relays both holographic objects and images of real-world objects ina way that allows for occlusion handling. FIG. 22 is the configurationof FIG. 11A with relay system 5100. The numbering of FIG. 11A is used inFIG. 22 . The optical fold system 1150 receives light rays 131A and 132Afrom holographic object surfaces 121A and 122A, respectively, andincreases the path length of these rays as the light rays continue todiverge into light rays 131B and 132B, respectively. An opticalcombining system comprising a beam splitter 101 combines the light rays131B and 132B from the optical fold system 1150 and the light rays 133Afrom the surface of the real-world object 123A, wherein some light rays133A may be partially or fully occluded by an occlusion system 150,which in an embodiment, may include a plurality ofindividually-addressed occlusion regions 188 on one or more occlusionlayers 151, 152, and 153. As described above, these layers 151, 152, 153may be transmissive OLED panels or a portion of LCD panels, and theindividually-addressable elements may be configured to be completelyopaque, semi-transparent, or substantially transparent. Some portion ofthe light rays 131B and 132B from holographic object surfaces 121A and122A, respectively, is reflected by the beam splitter 101 toward therelay system 5100 as light rays 131C and 132C, and these light rays arerelayed by relay system 5100 into converging light rays 131D and 132D,which form relayed holographic image surfaces 121B and 122B,respectively. The display surface 1021A is relayed into virtual displayplane 1022A. The operation of the relay system 5100 is described abovein reference to FIG. 21C. A portion of the light rays 133A from thereal-world object 123A pass through the image combiner 101, and then arerelayed to light rays 133B forming the relayed real-world image surface123C. As described above, occlusion regions 188 may result in no lightrays from the portion 189 of relayed real-world image surface 123C to bevisible behind relayed holographic image surface 121B as viewed by anobserver 1050, for an observer 1050. In this way, relayed holographicimage surface 121B appears to occlude the relayed background imagesurface 123C of real-world object 123A, just as it would if relayedholographic image surface 121B were a real physical object. In theembodiment shown in FIG. 22 , the angular filter 124 absorbs rays oflight 133R from the real-world object 123A that have an angle withrespect to the normal to the surface of the angular filter 124 thatexceeds a threshold value.

The relay system 5100 shown in FIG. 22 may result in a reversal of thedepth profile of the holographic object surfaces 121A and 122A when itrelays them to relayed holographic image surfaces 121B and 122B. Thiscan be corrected computationally using the reversal of u-v angular lightfield coordinates shown in FIGS. 2B and 2C. However, the relay system5100 also reverses the depth profile of the real-world object 123A whenrelaying an image of this object to form the relayed image surface 123C,and it may be very difficult or impossible to construct a real-worldscene 123A, which has a compensating reversed depth profile. Anotherapproach, as discussed previously in this disclosure, is to reverse thedepth of the real-world object by replacing the real-world object 123Awith a relayed depth-reversed image of the same object.

FIG. 23 illustrates the display system configuration of FIG. 22 , butthe real-world object 123A in FIG. 22 has been replaced with a relayedimage surface 123B of a real-world object 123A, using an input relaysystem 5030, which in an embodiment, may include a transmissivereflector. The numbering of FIG. 22 applies to FIG. 23 . FIG. 22 is alsothe configuration of FIG. 11A with relay system 5100, and wherein thereal-world object 123A is relayed twice. In FIG. 23 , light 133X fromthe surface of real-world object 123A is relayed to form thedepth-reversed relayed image 123B of real-world object 123A by relay5030. The depth-reversed relayed image 123B of real-world object 123A isonce again relayed by relay 5100 to relayed image of a real-world object123C with the same depth profile as real-world object 123A. As a result,the relayed surface of a real-world object 123C observed by viewer 1050has the same depth profile as the true real-world object 123A. The oneor more occlusion layers 150, 151, and 152 are disposed in front of thereal-world object, and after being relayed by relay 5030 and then relay5100, the relayed occlusion planes will be located between thetwice-relayed surface 123C of a real-world object and the observer 1050.Addressable regions 188 on these occlusion layers may be activated toblock out a portion of the light from real-world object 123A so thatlight from a corresponding occluded portion 189 of the relayed surface123C of the real-world object will not be visible behind a foregroundrelayed surface of a holographic object such as 121B for viewers 1050 inthe viewing volume of the relayed surfaces 121B, 122B, and 123C. Acontroller 190 may issue display instructions to the light field 1001Aand simultaneously issue occlusion instructions to the occlusion layers151, 152, and 153 in order to achieve the occlusion properly. Theup-down flip of the image 123C relative to the real-world object 123Amay be corrected by rotating the real-world object 123A or the use ofone or more mirrors. In the embodiment shown in FIG. 23 , the angularfilter 124 absorbs rays of light 133R from the real-world object 123Athat have an angle with respect to the normal to the surface of theangular filter 124 that exceeds a threshold value.

It is possible to use a simple lens system as a relay. FIG. 24 shows adisplay system which achieves simultaneous relay of both holographicobjects and images of real-world objects using a relay system 5070system comprised of one or more lenses 446 and 447. The relay system5070 is introduced earlier in this disclosure in reference to FIG. 4E.FIG. 24 is the configuration shown in FIG. 11B with the relay 5070utilized. The numbering of FIG. 23 is used in FIG. 24 for similarelements. In FIG. 24 , light 131A and 132A from holographic objectsurfaces 121A and 122A, respectively, is combined with light 133Y fromthe surface of a real-world object 123A by an optical combining system101, which may comprise a beam splitter, and the combined light isreceived by a relay system 5070 comprised of one or more lenses 446 and447. The lenses 446 and 447 may be concave lenses, convex lenses,diffractive lenses such as Fresnel lenses, or any other type of simpleor compound lenses. In FIG. 24 , the focusing effect of only one Fresnellens 446 is shown. The light rays 131A and 132A from holographic objectsurfaces 121A and 122A, respectively, are focused by the lens system5070 to converging light rays 131C and 132C which form relayedholographic image surfaces 121B and 122B, respectively, at relaylocations distributed around the relayed virtual screen plane 1022A. Thelight rays 133A are focused by lens relay 5070 to light rays 133C whichform the relayed image surface 123B of real-world object 123A. Anocclusion system 150, which may include one or more occlusion regions188 on occlusion planes 151, 152, and 153, may act to block out thelight rays from a portion 189 of relayed real-world image surface 123Bfrom reaching the observer 1050 when the observer 1050 is viewingrelayed holographic image surface 121B, so that relayed holographicimage surface 121B appears to be a real object occluding the relayedreal-world image surface 123B. To increase the optical path length oflight rays travelling through relay system 5070, and change the locationof the relayed holographic image surfaces 121B and 122B, as well as thelocation of the relayed image 123B of the real-world object 123A,optical folding systems 1150 (or 1160, 1170) may be placed either beforethe relay 5070 at 1150A, or after the relay 5070 at 1150B. An opticalfolding system such as 1150, 1160, or 1170 may be placed in the path ofthe light rays 133Y from the surface of real-world object 123A in orderto allow the real-world object 123A and the occlusion planes to becloser to the beam splitter 101 for a more compact design.

Relay systems which preserve a depth profiles are able to transport toanother location scenes presented by a stereoscopic, autostereoscopic,or multi-view displays, objects projected by a volumetric 3D display,holographic objects projected by a light field display, real-worldobjects emitting light, and real-world objects reflecting as they areoriginally exist, or as they are originally projected before beingrelayed. FIGS. 9A and 9G present a relay system comprised of twoseparate relays, in which the depth profile reversal of the first relayis substantially undone by the depth profile reversal of the secondrelay. It is possible to construct an imaging wherein light paths froman object are relayed twice by the same relay. Even if the relay invertsthe depth profile of an object during each pass of the relay, two passesthrough the relay will restore the depth profile of the object. Suchconfigurations may have the advantage of relaying an object withoutdepth reversal and may be economical in materials and size. FIG. 25A isan orthogonal view of a display system comprising a relay system 5110 inwhich the light from at least one object is relayed by passing throughthe same relay twice by reflecting from one or more mirrors. FIG. 25A isthe display system of FIG. 11B with the relay system 5110 utilized inplace of 5001.

The optical combining system 101 includes a first input interfaceconfigured to receive light along paths 131A from first image source1001 forming image surface 121A and a second input interface configuredto receive light along paths 133A from second image source 123A. Theconfiguration of FIG. 25A is the configuration of FIG. 11B with relay5110 utilized, where relay 5110 is comprised of a transmissive reflector5030 and two mirrors 2510A and 2510B. As described above in reference toin FIGS. 11A-D, the at least one of the first 1001 and second 123A imagesources may comprise: a 2D display, a stereoscopic display, anautostereoscopic display, a multi-view display in one axis (e.g. ahorizontal parallax only or HPO display), a volumetric 3D display, alight field display surface, a real-world object emitting light, areal-world object reflecting light, or the relayed image of a surface.In the example drawn in FIG. 25A, for the present discussion the firstimage source is a light field display 1001 operable to defineholographic image surface 121A and the second image source 123A may be a2D display with a 2D display surface or real-world object with areflective or emissive surface. The light rays combined by the imagecombiner 101 received by the relay 5110 include light rays 131A from thefirst surface of the holographic object 121A projected by the firstimage source light field display 1001 and deflected into light rays 131Bby 101, and the light rays 133A from the second surface of a 2D displayor real world object 123A which pass through the image combiner 101.Light rays 133A from the display or real-world object 123A are relayedinto light rays 133B focused toward a virtual convergence point 2511A.Light rays 133B reflect from the first mirror 2510A into light rays133C, which converge at first virtual display plane 123B, which is therelayed surface of the 2D display or real world object 123A. Light rays133C continue, reflecting from the second mirror 2510B into light paths133D. Light paths 133D diverge from virtual convergence point 2511B.These light rays 133D are received again by relay 5030 and are relayedinto light paths 133E, which converge to form a second virtual displayplane 123C, which is the twice-relayed surface of the 2D display or realworld object 123A. The light rays 131B from the holographic object 121Aare not shown to be relayed during intermediate steps shown in FIG. 25A,but these light paths are relayed by the relay shown in FIG. 25A in muchthe same way as light rays 133A from the display or real-world object,being relayed into light rays 131C which form relayed holographic imagesurface 121B. The one or more occlusion planes 151A may be a portion ofLC display panels, transmissive LED or LED panels, or some other type ofpanels with individually addressable occlusion sites 188. The distancebetween the one or more occlusion planes 151A from the display orreal-world object 123A may be selected so that the corresponding relayedocclusion plane 151B coincides with the relayed holographic object 121B,as shown in FIG. 25A. To arrange this, the distance between the one ormore occlusion planes 151 and the 2D display or real-world object 123Ashould be adjusted so that occlusion plane 151A and the projectedholographic object surface 121A are equidistant from the image combiner101, so that the relayed surface 123C of 2D display or real-world object123A may be occluded from being seen behind the relayed holographicimage surface 121B by an observer 1050 in as natural a way as possible(see FIGS. 9B, 9C, and 9D). This may be done to provide the correctdepth cues to viewer 1050 that the relayed holographic image surface121B is in front of the virtual object plane 123C. A controller 190 maygenerate display instructions for the light field display 1001 as wellas send configuration instructions to the one or more occlusion planes151A. In another embodiment, as shown in the configuration of FIG. 9B,it is possible that the one or more occlusion planes 151A will berelayed to virtual occlusion plane 151B at a location substantiallydifferent from the relayed holographic image surface 121B, but yet willstill provide effective occlusion for observers 1050. In anotherembodiment, the holographic display 1001 is swapped with the object 123Aand vice-versa in FIG. 25A, wherein the relayed object plane would beseen in front of the relayed holographic object, and the holographicobject may be occluded from being seen directly behind portions of therelayed object plane. In another embodiment, in FIG. 25A the light rays131A from the holographic object 121A may be combined with light rays133B, 133C, or 133D from object 123A by an image combiner placed betweenthe two mirrors 2510A and 2510B, allowing the object 123A to bepositioned closer to the transmissive reflector relay 5030. In thisconfiguration, the light from the holographic object 131A may reflectfrom one or both of mirrors 2510A-B in FIG. 25A, and this light 131A mayonly be relayed by one pass through the transmissive reflector 5030. Inanother embodiment, the two mirrors 2510A and 2510B may be replaced bythree mirrors in a 3-sided rectangular or square configuration whereinthe three sides of the mirrors may be orthogonal to one another and thefourth side of the rectangle or square is formed by the transmissivereflector 5030. In another embodiment, two or more mirrors may be usedin a different configuration to that shown in FIG. 25A to relay thelight from an object by passing the light multiple times through thesame relay. An embodiment with a transmissive reflector and a singlemirror is described next.

FIG. 25B is comprised of two orthogonal views of a display system with arelay system 5120 in which the light from at least one object is relayedby passing through the same relay twice by reflecting from a mirror. Theoptical combiner 101C includes a first input interface configured toreceive light along paths 131A from image source 1001 forming objectsurface 121A, and a second input interface configured to receive lightalong paths 133A from second image source 123A. The configuration ofFIG. 25B is the configuration of FIG. 11B with relay 5120 utilized,where relay 5120 is comprised of a transmissive reflector 5030, a mirror2510C, and a beam splitter 101D. As described above in reference to inFIGS. 11A-D, the at least one of the first 1001 and second 123A imagesources may comprise: a 2D display surface, a stereoscopic displaysurface, an autostereoscopic display surface, a multi-view displaysurface which may be the surface of a horizontal parallax-only HPOmulti-view display such as a lenticular display, the surface or surfacesof a volumetric 3D display, a light field display surface, the surfaceof a real-world object emitting light, or the surface of a real-worldobject reflecting light. In the example drawn in FIG. 25B, for thepresent discussion the first image source is a light field display 1001operable to define holographic image surface 121A and the second imagesource 123A may be a 2D display with a 2D display surface or real-worldobject with a reflective or emissive surface. The side view 2501 in FIG.25B reveals that the light rays received by the image combiner 101Cinclude the group of light rays 131A from the first surface of theholographic object 121A projected by the first image source light fielddisplay 1001, and the group of light rays 133A from the second imagesource 2D display or real-world object 123A. The light rays 131A formingthe holographic object 121A include light ray 1310A which is deflectedby image combiner 101C into light ray 1310B. The light rays 133A fromthe 2D display or the real-world object 133A include light ray 1330A and1331A projected at different angles, where light rays 1330A and 1331Aare combined with light ray 1310B and are received by the beam splitter101D of the relay system 5120, and these light rays 1330A, 1331A, and1310B are deflected into light rays 1330B, 1331B, and 1310C,respectively, by beam splitter 101D of the relay system 5120.

The top view 2502 in FIG. 25B shows how the light ray 1310C from theholographic object 121A and the light rays 1330B and 1331B from the 2Ddisplay or real-world object 123A traverse the relay system 5120. Thelight ray 1310C is relayed into light ray 1310D by transmissivereflector 5030, whereupon 1310D reflects from the mirror 2510C at thesame angle of approach into light ray 1310E which is relayed once againby the transmissive reflector 5030 into light path 1310F whichcontributes to forming the surface of relayed holographic object 121B.Similarly, 1330B and 1331B are relayed by the transmissive reflector5030 into light paths 1330C and 1331C, respectively, toward the mirror,reflecting from the mirror into light paths 1330D and 1331D which arethen relayed by the transmissive reflector 5030 into light paths 1330Eand 1331E which exit the relay 5120 by passing through beam splitter101D, and converge to form the relayed object 123B which may be therelayed surface of a 2D display 123A or the relayed surface of areal-world object 123A. In FIG. 25B, one or more occlusion planes 151Amay occlude a portion of the light from the object 2511A at occlusionsites such as 188, in order to block light from the portion of therelayed surface 123B of the 2D display or real-world object behind arelayed holographic image surface 121B from reaching an observer 1050. Acontroller 190 may generate display instructions for the light fielddisplay 1001 as well as send configuration instructions to the one ormore occlusion planes 151A. In FIG. 25B, the holographic object 121A iscloser from the first image combiner 101C than the 2D display orreal-world object 123A, and the corresponding relayed object 121B iscloser to the viewer 1050 than the relayed object 123B. As a result,depth may not be reversed by this relay 5120. FIG. 25B may have anoptional optical element 1041A located between the transmissivereflector 5030 and the reflective element 2510C, which may be a quarterwave retarder. If a polarization beam splitter 101D is used, then mostof the light 1330B, 1331B, and 1310C received by the relay 5030 andrelayed to respective light paths 1330C, 1331C, and 1310D toward thereflective element 2510C may be of a first polarization state. Thecombination of a quarter wave retarder 1041A and a reflective surface2510C may change these light paths to a state of second polarizationorthogonal to the first as they are again received by the relay 5030 andrelayed through the beam splitter 101D whereupon most of these lightrays will pass without being deflected. This may result in less lightloss for the relay system 5120.

FIG. 25C is an orthogonal view of an imaging relay system 2503 comprisedof a transmissive reflector 5030 with a polarization beam splitter 2521on one side of the transmissive reflector, and a mirror 2510D pairedwith a quarter wave retarder 2522, the plane of the mirror disposed atan acute angle relative to the surface of the transmissive reflector5030. The plane of the polarization beam splitter 2521 is placedparallel to the face of the transmissive reflector 5030, on the side ofthe mirror, with the polarization beam splitter 2521 possibly attachedto the surface of 5030. The polarization beam splitter 2521 may pass afirst state of linear polarization and reflect a second state of linearpolarization orthogonal to the first. In some embodiments, thepolarization beam splitter 2521 may pass a first state of circularpolarization and reflect a second state of circular polarizationorthogonal to the first. In some embodiments the quarter wave retarder2522 is another polarization element, such as a half wave plate, or maybe absent altogether. The plane of the quarter wave retarder 2522 isdisposed to be parallel to the plane of the mirror 2510D, on thereflective part of the mirror, and may be attached to the plane of themirror. In one embodiment, the angle between the mirror 2510D and thetransmissive reflector 5030 is about 22.5 degrees, but otherconfigurations with different angles may be achieved. Incident lightrays of a first linear polarization state to the relay system 2503 alongpath 1, designated by the solid line, are received by the transmissivereflector 5030, and relayed into path 2, passing through thepolarization beam splitter 2521 and toward the mirror 2510D. Beforereaching the mirror 2510D along path 2, the quarter wave retarder 2522changes the polarization state of the light 2 from a first polarizationstate into a first circular polarization state. Upon reflection of thislight 2 from the mirror into path 3, the first circular polarizationstate is converted into a second circular polarization state orthogonalto the first. After passing again through the quarter wave retarder2522, the light on path 3 is converted into a second state of linearpolarization orthogonal to the first state of linear polarization onpath 2, designated by the dashed line along path 3. In other words, thelinear state of polarization of path 2 has been converted from a firstto a second state upon a first pass through quarter wave retarder 2522,reflecting from mirror 2510D, and passing a second time through thequarter wave retarder 2522, which is well known in the art. The light onpath 3 of the second state of linear polarization is reflected from thepolarization beam splitter 2521 into path 4 without changing state, sothe line for path 4 in FIG. 25C is shown as remaining dashed. Uponreflection of path 4 from the mirror, the second state of linearpolarization of path 4 changes into a first state of linear polarizationfor path 5, which is shown as a solid line. This state of polarizationmay pass through the polarization beam splitter 2521, and so path 5 isrelayed into path 6 by the transmissive reflector where path 6intersects with path 1 at point 25115. This point of intersection 25115for an incident light ray may be adjusted by changing the distance 25114between the mirror 2510D and the transmissive reflector 5030. The relaysystem 2503 is reciprocal—in the example of FIG. 25C, light input onpath 1 is relayed into path 6, but light input on path 6 will be relayedinto path 1. This means light from a point 25115 received by the relaysystem 2503 will return to that point with the light ray angles swapped.

FIG. 25D is an orthogonal view of the light paths generated within therelay system shown in FIG. 25C for three input angles of light from apoint source. Light input at three angles along light paths 25117A,25118A, and 25119 pass through common point 25116, are received by therelay, are reflected, and exit the relay along paths 25117B, 25118B, and25119, respectively. Light input along the center path 25119 returnsalong this same center path but with the direction reversed. A light rayalong path 25117A received by relay 2503 at an incident angle −φrelative to this center path 25119 is returned along a path 25117B at 9,the negative of the incident angle.

FIG. 25E is a display system employing the relay system 2503 shown inFIG. 25C to relay an object 2521A to a relayed object 2521B. Light rays2550, including light rays along light paths 2522A, 2532A, and 2542A aredirected toward an image combiner 101E. Light path 2522A is reflected bythe image combiner 101E into path 2522B, which is received by the relaysystem 2503 and relayed to light path 2522C, which passes through theimage combiner 101E. Similarly, light path 2532A is reflected by imagecombiner 101E into path 2532B, which is received by relay system 2503and relayed to light path 2532C, which passes directly through the imagecombiner 2503. The vertical light path 2542A leaving object 2521A, isreflected by the image combiner 101E, received by the relay system 2503along light path 2542B in a direction toward the relay system 2503,relayed back along light path 2542B in the opposite direction away fromthe relay system 2503, and straight through the image combiner 101E. Therelayed light paths 2522C, 2532C, and 2542B converge to form the relayedobject 2521B. In FIG. 25E, the desired distance 2525 between the relaysystem 2503 and the relayed object position 2521B may be tuned byadjusting the distance 25114 between the mirror 2510D and thetransmissive reflector 5030 shown in FIG. 25C. The distance between theobject 2521A and the image combiner 101E may be set equal to thedistance between the relayed object 2521B and the image combiner 101E.In an embodiment, object 2521A may be replaced by any of: a 2D displaysurface, a stereoscopic display surface, an autostereoscopic displaysurface, or a horizontal parallax-only multi-view display such as alenticular display.

Motion of Relayed Holographic and Real-World Objects

This disclosure has presented a number of ways to combine holographicobjects with images of real-world objects in such a way that they appeartogether in approximately the same location, and occlusion of theholographic objects overlapping with the image of the real-world objectsmay be handled with the use of occlusion barriers. There are severalways in which motion of the holographic objects or real-world objectsmay be handled, which are outlined below.

FIG. 26A is the same display system shown in FIG. 11A in which relaysystem 5000, but with arrows showing how relayed holographic objectsurfaces 121B and 122B may be moved computationally. The relay 5000relays light from holographic object surfaces projected from a firstimage source light field display 1001A simultaneously with the lightfrom second image sources of one or more real-world objects, summarizingmany of the systems shown in FIGS. 9A and FIGS. 11-24 . The numbering ofFIG. 11A applies to FIG. 26A. The relay system 5000 is shown to reversethe depth profile of relayed objects (e.g. relayed holographic objectsurfaces 121B and 122B have a reverse depth profile from the projectedobject surfaces 121A and 122B), but the discussion here also applies toa display system shown in FIG. 11B with relay 5001 which preserves thedepth ordering of surfaces that are relayed. The discussion shown inFIG. 26A also applies to the variations shown in FIGS. 11D and 11E inwhich the first and second image sources each comprises: a 2D displaysurface, a stereoscopic display surface, an autostereoscopic displaysurface, a multi-view display surface which may be the surface of ahorizontal parallax-only HPO multi-view display such as a lenticulardisplay, the surface or surfaces of a volumetric 3D display, a lightfield display surface, the surface of a real-world object emittinglight, or the surface of a real-world object reflecting light. In anembodiment, the relay system may include a controller 190 configured tosupply display instructions to the light field display 1001A and the oneor more occlusion planes 151, 152, and 153. FIG. 26A demonstrates howholographic objects may be moved completely computationally. In FIG.26A, the holographic object surface 121A is moved in a direction denotedby the arrow A by the controller 190 supplying display instructions tothe display 1001A. The display instructions may be determined from arendering engine. The controller 190 may also issue instructions to anocclusion system 150, which in an embodiment, may include the occlusionplanes 151, 152, and 153, to provide the correct real-time occlusionregions 188 to occlude light rays from real-world object 123A such thatfor possible viewing locations for observer 1050, the portion 189 of therelayed image surface 123B of the real-world object 123A that is behindthe moving relayed holographic image surface 121B does not transmitlight. Occlusion regions 188 move in the direction denoted by the arrowA near 188, and in turn, the occluded portion 189 of the relayed imagesurface 123B will move in a direction denoted by the arrow A near 189.All of this movement is achieved computationally. In an embodiment, anoptical system comprises a controller 190 operable to coordinate amovement of the occlusion region 188 with a movement of an image surface121B or 122B in the viewing volume.

In an embodiment, the occlusion barriers 151, 152, and 153 in FIG. 26Amay be replaced with at least one real-world occlusion object. In anembodiment, the at least one occlusion object may be configured to havethe same dimensions as a relayed holographic object 121B, 122B and ismoved mechanically in synchronization with movement of the holographicobject, wherein the holographic object may be moved computationally.FIG. 26B is the display system of FIG. 26A with areal-world object 121ASreplacing the occlusion barriers 151, 152, and 153 in the occlusionsystem 150 shown in FIG. 26A. The numbering in FIG. 26A is used in FIG.26B. The real-world object 121AS is designed to be an occlusion object,which may be painted matte black or have a light-absorbing texture andhas a position which is motor controlled. In FIG. 26B, holographicobject surface 121A is moved to the left along arrow B near 121A viadisplay instructions from the controller 190. In response, relayedholographic image surface 121B moves vertically along arrow B near 121Bin response to holographic object surface 121A being moved. The object121AS may be motorized in an embodiment, and the controller 190 may alsoissue instructions to a motor, which moves occlusion object 121AS in thedirection along arrow B near 121AS. The moving motorized occlusionobject 121AS blocks light rays leaving real-world object 123A, allowingthe occluded portion 189 of the relayed real-world image surface 123B tomove vertically along the arrow B near 189, moving to track the motionof the relayed holographic image surface 121B, so that the relayedholographic image surface 121B seems to occlude the relayed backgroundimage surface 123B of real-world object 123A. In an embodiment, at leastone occlusion object 121AS is motorized. In a further embodiment, theoptical system comprises a controller 190 operable to coordinate amovement of the at least one occlusion object 121AS with a movement ofan image surface 121B or 122B in the viewing volume.

In an embodiment, motion of both the relayed holographic image surfaces121B and 122B, as well as the relay image surface 123B of the real-worldobject can be moved by simply mechanically moving the relay system 5000,or a portion of the relay system 5000. FIG. 26C is the display system ofFIG. 26A showing the direction of motion for many of the elements shownin FIG. 26A when the relay system 5000 is moved vertically alongdirection of arrow C near relay 5000. The numbering of FIG. 26A is usedin FIG. 26C. This motion of the relay 5000 results in both an upwardmotion for the relayed images 121B, 122B, and 123B, as well as therelayed images being projected further, for a combined motion diagonallyupward toward the top left of the page along the associated arrows Cnear relayed objects 121B, 122B, and 123B. Depending on whichconfiguration of the relay system 5000 is used, under some circumstancesthe controller 190 may issue instructions to the occlusion layers 151,152, and 153 to adjust the occlusion regions 188, denoted by thedownward arrow C, so that the occluded portion 189 of the relayed imagesurface 123B of the real-world object 123A tracks the motion of therelayed holographic object image surface 121B, so that the relayedholographic image surface 121B continues to appear to occlude therelayed image surface 123B of real-world object 123A. In an embodiment,a relay system 5000 comprises a mechanical mechanism operable to imparta motion of the relay system relative to at least one occlusion layer151, 152, or 153 and the first and second image sources 1001A and 123A,wherein the relay system moves relative to the rest of the opticalsystem. In another embodiment, the relay system 5000 comprises acontroller operable to coordinate a movement of the relay system with amovement of an image surface 121B, 122B defined in the viewing volume,so that the desired movement of the relayed image surface may beachieved. In still another embodiment, a relay system comprises acontroller 190 operable to coordinate a movement of the relay system5000 with a movement of an occlusion region 188 defined by the at leastone occlusion layer 151, 152, or 153 in order to allow for adjustableocclusion handling of relayed objects 121B, 122B, and 123B as they movein response to the relay movement. The optical display system shown inFIG. 26C may have an occlusion system comprised of a real-worldocclusion object like 121AS shown in FIG. 26B. In an embodiment, therelay system 5000 comprises a mechanical mechanism operable to impart amotion of the relay system relative to the at least one occlusion object121AS and the first and second image sources 1001A and 123A, and acontroller 190 is operable to coordinate a movement of the relay system5000 with the movement of the at least one occlusion object in order tocorrectly account for occlusion as the relayed objects 121B, 122B and123B move in response to the relay motion. In still another embodiment,the relay system comprises a mechanical mechanism operable to impart amotion of the relay system 5000 relative to the at least one occlusionobject 121AS and the first and second image sources 1001A and 123A, anda controller 190 is operable to coordinate a movement of the relaysystem with the movement of an image surface 121B, 122B, and 123B in theviewing volume.

FIG. 26D is the display system of FIG. 26A showing three other optionsD, E, and F for motorized movement of some of the components of therelay system 5000. The numbering of FIG. 26A is used in FIG. 26D. Inoption D, the light field display 1001A is moved by a motor upward indirection D. In response, the relayed holographic image surfaces 121Band 122B move to the right, along arrows D near these objects. In anembodiment, at least one of the first and second image sources 1001A and123A is movable to impart motion relative to the at least one occlusionlayer. In another embodiment, at least one of the first and second imagesources 1001A and 123A is movable to impart motion relative to the atleast one occlusion object. In option E, the real-world object 123A ismoved by a motor downward in the direction of arrow E near 123A, butnothing else is moved. In response, the relayed image surface 123B ofthe real-world object 123A moves upward along arrow E near 123A, but therelayed holographic image surfaces 121B and 122B do not move. Lastly, inoption F, all the hardware components including the light field display1001A, the relay system 5000, the optical combining system 101, thereal-world object 123A, the optical folding systems 1150, and theocclusion barriers 151, 152, and 153 of the occlusion system 150 movewith a motor along direction F. This causes the relayed holographicimage surfaces 121B, 122B, and the relayed real-world image surface 123Bto move relative to a stationary observer 1050 along the arrows F shownnext to these respective objects. Finally, although not illustrated inFIG. 26A-D, it is possible to adjust an occlusion layer or an occlusionobject by simply moving the occlusion layer or object. In an embodiment,the movement of the occlusion region 188 in the at least one occlusionlayer 152 is effected at least in part by a physical motion of the atleast one occlusion layer. In an embodiment, the occlusion region in theat least one occlusion layer is effected at least in part by modulatingindividually addressable elements in the at least one occlusion layer.

The motions shown in FIG. 26A-D are exemplary motions in particulardirections, and many other directions of motion are possible for theelements of the display system 26A. As stated earlier, otherconfigurations of display systems shown in FIGS. 11A-H or any otherdisplay system with relays presented in this disclosure may move relayedobjects in a similar manner. Depending on the configuration of the relay5000 or any other relay used in the display system, the motionsdescribed here may be accompanied by minification or magnification of aprojected holographic object surface, a computational swap of U-Vcoordinates in order to reverse depth, or the computational adjustmentof U-V mapping for light rays forming projected holographic objectsurfaces in order for the corresponding relayed objects to appear tomove smoothly and without distortion. Finally, although this discussionhas focused on a first image source of a light field display and asecond source of a real-world object with an emissive or reflectivesurface, the first and second image sources may include a 2D displaysurface, a stereoscopic display surface, an autostereoscopic displaysurface, a multi-view display surface which may be the surface of ahorizontal parallax-only HPO multi-view display such as a lenticulardisplay, the surface or surfaces of a volumetric 3D display, a lightfield display surface, the surface of a real-world object emittinglight, or the surface of a real-world object reflecting light, asdetailed above in the discussion for FIGS. 11A-11I and the other displayconfigurations of this disclosure which comprise at least one imagerelay.

Multi Relay Display Systems

Often imaging relay systems are more limited in field-of-view (FOV) thandesired for a display application. For example, the FOV of atransmissive reflector or a retroreflector is about 45 degrees (+/−22.5degrees), which means that a relay system built from such components maybe limited to this output range of angles. To overcome this limitation,it is useful to use configurations with multiple relay systems. FIG. 27Ais an orthogonal view of the surfaces of two relays angled with respectto one another to create a combined field-of-view (FOV) which is largerthan either of the FOVs of the individual relays. Only the exit surface2701A and 2701B of each relay is shown. While the surface is shown to bean angled surface, which could be the angled image combiner or theangled transmissive reflector of relays such as those illustrated inFIGS. 9A, 9G, FIG. 12-19 , or FIG. 25A, 25B or 25E, the surface could beplanar, similar to the relays illustrated in FIG. 20 and FIG. 24 . Afirst relay 2701A may have a range of output angles for relayed lightpaths 2702A limited by a first FOV 2703A, while a second relay 2701B mayhave a range of output angles for relayed light paths 2702B limited by asecond FOV 2703B. However, if the first relay surface 2701A and secondrelay surface 2701B are disposed next to one another, and in thisconfiguration rotated with respect to one another by angle 2704, then acombined FOV 2703C may be achieved wherein a light path from either thefirst relay 2701A or the second relay 2701B may be observed at everyangle. In an embodiment, the viewing volume of the relay system 2701Adefines a first field of view 2703A; wherein the optical system furthercomprises an additional relay system 2701B configured to relay lightfrom at least one additional image source along light paths to anadditional viewing volume that defines a second field of view 2703B, andwherein the first relay system 2701A and the additional relay system2701B are aligned such that the first and second fields of view arecombined to define a combined field of view 2703C.

FIG. 27B is an orthogonal view of an implementation of the concept shownin FIG. 27A, comprising two identical display systems 1400 shown in FIG.14A, each display system 1400 configured with a transmissive reflectorrelay, wherein the two display systems are arranged so that the FOV forthe relayed objects is larger than the FOV for either of the separatedisplay systems 1400. The relays 5030 and 50300 from the two displaysystems 1400A and 1400B, respectively, are disposed at an angle 2704with respect to one another. In one embodiment, the angle 2704 is lessthan 90 degrees. In FIG. 27B, A the numbering of FIG. 14 is used for thefirst display system 1400A, and the discussion of FIG. 14A describes indetail how objects are relayed within this display system. Within thefirst display system 1400A, relay 5030 relays a projected holographicobject 121A to the relayed object 121B and projected holographic object122A to relayed holographic object 122B. The surface of a real-worldobject 123A is relayed to surface 123B via transmissive reflector 5030A,and surface 123B is relayed to relayed surface 123C of real-world object123A via transmissive reflector 5030. Similarly, within the seconddisplay system 1400B, relay 50300 relays projected holographic surface1210A to relayed holographic object 1210B and projected holographicobject 1220A to relayed holographic object 1220B. The surface ofreal-world object 1230A is relayed to relayed surface 1230B viatransmissive reflector 50300A, and surface 1230B is relayed to relayedsurface 1230C of real-world object 1230A via transmissive reflector50300. Note that as pictured, the twice relayed real-world images ofobjects 123C from the first relay and 1230C from the second relay do notoverlap. Moreover, these two relayed objects are up-down flipped. Toachieve alignment between these relayed objects from the two relays, atseveral adjustments may be made. The first adjustment is to rotatetoward one another 2706A and 2706B the image combining systems 1205A and1205B within each relay system 1400A and 1400B, respectively, each imagecombining system comprised of all the optical components in each relaysystem except for the transmissive reflector. The image combining system1205A comprising the first relay system 1400A may be rotatedcounterclockwise 2706A, and the image combining system 1205B comprisingthe second relay system 1400B may be rotated clockwise 2706B. Inaddition, one of the real-world objects 1230A within one of the relaysystems 1400B may be rotated 1208 by an angle of about 180 degrees, butstill have its surface aligned substantially parallel to the occlusionlayers such as 1520A. The occlusion region 188B within relay system1400B should also move in coordination with the movement of thereal-world object 1230A. Also, to achieve vertical alignment between therelayed real-world objects 123C and 1230C, real-world object 123A maymove in the direction indicated by the arrow 1207A, and the real-worldobject 1230A may move in the direction indicated by the arrow 1207B. Theocclusion sites 188 within one or more occlusion planes 152 within thefirst relay system 1400A may adjust to the new position of real-worldobject 123A, while the occlusion sites 188B within one or more occlusionplanes 1520A within the second relay system 1400B may adjust to the newposition of real-world object 1230A. Similar adjustments in position tothe ones just described may be made to the projected holographic objects121A, 1210A, 121B, and 1210B. This example shown in FIG. 27B is only oneimplementation of several adjustments that may be made to one or morerelay systems described earlier in this disclosure to achieve a combinedFOV. There are many other configurations with varying angles of imagecombiners relative to the relay systems, placement of displays orreal-world objects, projection of holographic objects, and otherconfigurations which achieve a combined FOV using more than one relaywhich is greater than the single FOV of a display system with a singlerelay.

FIG. 27C is an orthogonal view of the display system shown in FIG. 27Bwherein adjustments to each display system have been made to achieveoverlap of relayed objects. The display system 1401A is the displaysystem 1400A shown in FIG. 27B with some adjustments shown in FIG. 27Band described above including the rotation 2706A of the image-combiningsystem 1205A relative to the transmissive reflector 5030, movement 1207Aof the real-world object 123A to a new position, and some possiblereadjustment of the positions of projected holographic objects 121A and122A to new locations 121D and 122D, respectively. The display system1401B is the display system 1400B shown in FIG. 27B with someadjustments shown in FIG. 27B and described above including the rotation2706B of the image-combining system 1205B relative to the transmissivereflector 50300, translation 1207B and rotation 1208 of the real-worldobject 1230A to a new position, and some possible readjustment of thepositions of projected holographic objects 1210A and 1220A to newlocations 1210D and 1220D, respectively. Both display system systems1401A and 1401B are shown with a controller 190A and 190B, respectively,where 190A and 190B may be the same controller. In FIG. 27C, withinrelay system 1401A, light rays 1214A from a real-world object 123A arerelayed by a transmissive reflector 5030A to light paths 1214B. Lightpaths 1214B form relayed image 123D, and are reflected into light rays1214C by the image combiner 101, which combines these light paths 1214Cwith light 1220 from the holographic object 121D and light from theholographic object 122D (not shown for simplicity) projected from lightfield display 1001A. At this point in the drawing, only one ray 1216A ofthe group of light rays 1220 from the holographic object 121D is shownto continue through the image combiner 101 to avoid clutter of the FIG.27C. Light rays 1214C and light ray 1216A are shown to be received bythe relay 5030 and relayed to light rays 1214D and light ray 1216B,respectively, where relayed light rays 1214D form a portion of therelayed surface 1213 of real-world object 123A, and light ray 1216Bforms a portion of the relayed holographic object 1211. Note that lightpath 1216A is projected at an angle normal to the surface of the lightfield display 1001A at light field angular coordinate (u, v)=(0, 0), butthe corresponding relayed light path 1216B is not normal to the viewer1050, and therefore has a different light field angular coordinate than(u, v)=(0, 0). In this case the 4D light field coordinates produced bylight field display 1001A may be remapped computationally by thecontroller 190A so that the relayed holographic object 1211 has theappearance and depth profile intended for a viewer 1050. The one or moreocclusion planes 188C may be activated in order to block some of theunwanted light paths. For example, light path 1218A of the group oflight paths 1214A reflected or emitted by real-world object 123A andrepresented by the only dashed line in the group 1214A-D is relayed tolight path 1218B which helps form the relayed surface 1213 of real-worldobject 123A. It may be desired that observer 1050D looking at therelayed holographic object 1212 should not be able to see relayedreal-world object 1213 behind holographic object 1212. For this reason,the light ray 1218A may be blocked by an individually addressableocclusion region 188C on the one or more occlusion layers 152. Thecontroller 190A may generate display instructions for the light fielddisplay 1001A as well as send configuration instructions to the one ormore occlusion planes 152.

In FIG. 27C, within display system 1401B, light rays 1215A from areal-world object 1230A are relayed by a transmissive reflector 50300Ato light paths 1215B. Light paths 1215B form relayed image surface1230E, and these light paths are reflected into light rays 1215C by theimage combiner 101D, which combines these light paths 1215C with light1221 from the holographic object 1210D and light from the holographicobject 1220D (not shown for simplicity) projected from light fielddisplay 1001D. At this point in the drawing, only one ray 1217A of thegroup of light rays 1221 from the holographic object 1210D is shown tocontinue past the image combiner 101D in order to avoid cluttering theFIG. 27C. Light rays 1215C and light ray 1217A are shown to be receivedby the relay 5030D and relayed to light rays 1215D and light ray 1217B,respectively, where relayed light rays 1215D form a portion of therelayed image surface 1313 of a real-world object 1230A, and light ray1217B forms a portion of the relayed holographic object 1211. Note thatlight path 1217B is projected at an angle normal to the surface of thelight field display 1001D at light field angular coordinate (u, v)=(0,0), but the corresponding relayed light path 1217B is not normal to theviewer 1050, and therefore has a different light field angularcoordinate than (u, v)=(0, 0). In this case the 4D light fieldcoordinates produced by light field display 1001D may be remappedcomputationally by the controller 190B so that the relayed holographicobject 1211 has the appearance and depth profile intended for a viewer1050. The one or more occlusion planes 188D may be activated in order toblock some of the unwanted light paths. For example, in may be desirablefor observer 1050 looking at the relayed holographic object 1211 to notbe able to see relayed real-world object 1213 behind holographic object1211. For this reason, the source light rays 1215A may be blocked by oneor more individually-addressable occlusion regions 188D on the one ormore occlusion layers 1520A. The controller 190B may generate displayinstructions for the light field display 1001D as well as sendconfiguration instructions to the one or more occlusion planes 1520A.The controller 190B in relay system 1401B may be the same as controller190A in relay system 1401A and may send instructions to both light fielddisplays 1001A and 1001D in FIG. 27C, and both sets of the one or moreocclusion planes 152 and 1520A. The real-world object 123A may be aduplicate of real-world object 1230A.

Examining all the light paths in FIG. 27C, it is clear that both thedisplay systems 1401A and 1401B may contribute light rays to the sameimage of a relayed real-world object 1213 or the same relayed surfaces1211 or 1212 of projected holographic objects 121D/1210D or 122D/1220D.The FOV of light relayed by display systems 1401A and 1401B may each besimilar to the FOV 2703A and 2703B shown in FIG. 27A, while the combinedFOV of relayed object surfaces 1211, 1212, or 1213 may be similar to thewider angular range 2703C shown in FIG. 27A. In an embodiment, theviewing volume of the relay system 1401A defines a first field of view1229A; wherein the optical system further comprises an additional relaysystem 1401B configured to relay light from at least one additionalimage source along light paths to an additional viewing volume thatdefines a second field of view 1229B, and wherein the first relay system1401A and the additional relay system 1401B are aligned such that thefirst and second fields of view 1229A and 1229B are combined to define acombined field of view 1229C. In another embodiment, the at least oneadditional image source in additional relay 1401B comprises first andsecond additional image sources 1001D and 1230A, wherein the opticalsystem further comprises a third input interface configured to receivelight from the first additional image source 1001D and a fourth inputinterface configured to receive light from the second additional imagesource 1230A wherein the additional relay system is configured to directlight from the first and second additional image sources 1001D and 1230Ato the additional viewing volume defining the combined field of view1229C.

FIG. 27D is an orthogonal view of a relay system comprised of twoseparate relays 5040A and 5040B angled with respect to one another tocreate a combined field-of-view (FOV) which is larger than either FOV ofthe separate relays, where each relay 5040A and 5040B is relay 5040shown in FIG. 5D comprised of an image combiner and a curved mirror. Therelays 5040A and 5040B each have a relay input interface configured toreceive light. In one embodiment, the relays 5040A and 5040B eachreceive light along a set of light paths directly from at least a firstimage source, wherein the light from the first image source is operableto define at least one first image surface. The first image source foreach relay 5040A and 5040B may be a light field display, and the firstimage surface may be the surface of a holographic object projected bythe light field display. For example, 5040A and 5040B may each be relays5040 in the configuration shown in FIG. 5D which relays light from afirst light field display image source 1001 which projects holographicimage surfaces 1015C and 1016C. In another embodiment, the relays 5040Aand 5040B each receive combined image light from an optical combiningsystem comprising a first optical combining input interface configuredto receive light along a first set of light paths from a first imagesource wherein the light from the first image source is operable todefine a first image surface, and second optical combining inputinterface configured to receive light along a second set of light pathsfrom a second image source wherein the light from the second imagesource is operable to define a second image surface. As an example, eachrelay 5040A and 5040B may be the relay 5050 (with only one mirror) of adisplay system shown in FIG. 17 , where each relay 5050 receivescombined light from the optical combining system shown in FIG. 17comprising image combiner 101A which receives a first set of light paths131A and 132A from a first image source light field display 1001A whichprojects image surfaces of holographic objects 121A and 122A,respectively, as well as a second set of light paths 133A generated by areflective or emissive real-world object 123A image source that has areal-world object surface. While the examples of FIG. 5D and FIG. 17have been presented here with a first image source as a light fielddisplay for relays 5040A and 5040B, the first and second image sourcesmay each be any of: a 2D display surface, a stereoscopic displaysurface, an autostereoscopic display surface, a multi-view displaysurface which may be the surface of a horizontal parallax-only HPOmulti-view display such as a lenticular display, the surface or surfacesof a volumetric 3D display, a light field display surface, the surfaceof a real-world object emitting light, or the surface of a real-worldobject reflecting light. Correspondingly, the image surface of thesecond image source may include an image surface projected from a 2Ddisplay surface, an image surface projected from a stereoscopic displaysurface, an image surface projected from an autostereoscopic displaysurface, an image surface projected from a multi-view display surface,an image surface of a volumetric 3D display, a surface of a holographicobject formed by light paths projected from a light field display, asurface of a real-world object, or a relayed image of the surface of thereal-world object.

FIG. 27E is an orthogonal view of a relay system comprised of twoseparate relays 5100A and 5100B angled with respect to one another tocreate a combined field-of-view (FOV) which is larger than either FOV ofthe separate relays, wherein each separate relay 5100A and 5100B is therelay system 5100 shown in FIG. 20 comprised of a transmissiveretroreflector, a reflective surface, and one or more layers of optionaloptical layers which may include polarization filters, quarter waveretarders, half wave retarders, or the like, and described above inreference to FIG. 20 . The relays 5100A and 5100B each have a relayinput interface configured to receive light. In one embodiment, therelays 5100A and 5100B each receive light along a set of light pathsdirectly from at least a first image source, wherein the light from thefirst image source is operable to define at least one first imagesurface. The first image source for each relay 5100A and 5100B may be alight field display, and the first image surface may be the surface of aholographic object projected by the light field display. For example,5100A and 5100B may each be relays 5100 in a display system shown inFIG. 21C which relays light from a first light field display imagesource 1001A projecting holographic image surfaces 121A and 122A. Inanother embodiment, the relays 5100A and 5100B each receive combinedimage light from an optical combining system comprising a first opticalcombining input interface configured to receive light along a first setof light paths from a first image source wherein the light from thefirst image source is operable to define a first image surface, andsecond optical combining input interface configured to receive lightalong a second set of light paths from a second image source wherein thelight from the second image source is operable to define a second imagesurface. As an example, each relay 5100A and 5100B may be the relay 5100in the display system shown in FIG. 22 , where each relay 5100A and5100B receives combined light from the optical combining system shown inFIG. 22 comprised of image combiner 101 which receives a first set oflight paths 131B and 132B from a first image source light field display1001A which projects image surfaces of holographic objects 121A and122A, respectively, as well as a second set of light paths 133Agenerated by a reflective or emissive real-world object 123A imagesource that has a real-world object surface. While the examples of FIG.21C and FIG. 22 have been presented here with a first image source as alight field display for relays 5100A and 5100B, the first and secondimage sources may each be any of: a 2D display surface, a stereoscopicdisplay surface, an autostereoscopic display surface, a multi-viewdisplay surface which may be the surface of a horizontal parallax-onlyHPO multi-view display such as a lenticular display, the surface orsurfaces of a volumetric 3D display, a light field display surface, thesurface of a real-world object emitting light, or the surface of areal-world object reflecting light. Correspondingly, the image surfaceof the second image source may include an image surface projected from a2D display surface, an image surface projected from a stereoscopicdisplay surface, an image surface projected from an autostereoscopicdisplay surface, an image surface projected from a multi-view displaysurface, an image surface of a volumetric 3D display, a surface of aholographic object formed by light paths projected from a light fielddisplay, a surface of a real-world object, or a relayed image of thesurface of the real-world object.

FIG. 27F is an orthogonal top view of a combined display systemcomprised of two display systems 9002A and 9002B placed side by side,where each display system is the display system 9002 shown in FIG. 9G,wherein the combined display system has a combined FOV that is almosttwice the FOV of a single display system 9002. The display system inFIG. 27F is comprised of 9002B, which is an exact copy of the displaysystem 9002 shown in FIG. 9G, and display system 9002A, which is anexact copy of the display system 9002 shown in FIG. 9G but rotated 180degrees from the top view, and placed directly aside relay 9002B.

In an embodiment, the viewing volume of the relay system 9002A defines afirst field of view 2720A; wherein the optical system further comprisesan additional relay system 9002B configured to relay light from at leastone additional image source along light paths to an additional viewingvolume that defines a second field of view 2720B, and wherein the firstrelay system 9002A and the additional relay system 9002B are alignedsuch that the first and second fields of view 2720A and 2720B arecombined to define a combined field of view 2720C. In anotherembodiment, the at least one additional image source in additional relay9002A comprises first and second additional image sources light fielddisplay 1001F shown in FIG. 9G and object 123F, wherein the opticalsystem further comprises a third input interface configured to receivelight from the first additional image source 1001F and a fourth inputinterface configured to receive light from the second additional imagesource 123F wherein the additional relay system 9002B is configured todirect light from the first and second additional image sources 1001Fand 123F to the additional viewing volume defining the combined field ofview of 2720C.

The numbering of FIG. 9G applies to the numbering of FIG. 27F, and thediscussion of FIG. 9G above describes how light paths are relayed withineach of the display systems 9002A and 9002B to relay the surface of aprojected holographic object and the surface of a real-world object or adisplay, with the relayed background surface of the real-world object ordisplay possibly occluded by the relayed foreground surface of theprojected holographic object. The relay system 5090 in each displaysystem 9002A and 9002B is comprised of two transmissive reflectors withan image combiner between them: display system 9002A is comprised ofrelay system 5090A which is relay 5090 in FIG. 9G with paralleltransmissive relays 5030D, 5030E and image combiner 101, while displaysystem 9002B is comprised of relay system 5090B which is also theconfiguration of relay 5090 in FIG. 9G with parallel transmissive relays5030F, 5030G and image combiner 101. The combined relay 50901 of thecombined display system shown in FIG. 27F is comprised of side-to-siderelays 5090A and 5090B, which are disposed next to one another withoutput relay faces 5030E and 5030G forming an angle 2704A which may beless than 90 degrees just like the acute angle 2704 in FIG. 27A. Thecombined relay 50901 is comprised of four transmissive reflectors5030D-F arranged to form the side-to-side relays 5090A and 5090B.

In the top display system in FIG. 27F, the relay 5090B relays light raysfrom a projected holographic object (numbered 121F in the side view ofFIG. 9G, but not shown in this top-view diagram) to light rays 131J and133G which form relayed holographic object 121H. In this discussion, thereal-world object or display 123F will be called an object 123F.Similarly, light rays from object 123F are relayed to light paths 133Fwhich form the relayed surface 123H of the object 123F. The light rays131J and 133G forming the relayed holographic object 121H as well as thelight rays 133F forming the relayed object 123H are projected into theangular range 2720B and observed by observer 1050H. The one or moreocclusion planes 150F is relayed to relayed plane 150H. Similarly,within the bottom relay in FIG. 27F, the relay 9002A relays light raysfrom a holographic object projected from a light field display separatefrom the one in relay 9002B to light rays 1310J and 1330G which formrelayed holographic object 121H. Similarly, light rays from object 1230Fare relayed to light paths 1330F which contribute to forming the relayedobject 123H. The light rays 1310J and 1330G which contribute to formingthe relayed holographic object 121H as well as the light rays 133Fforming the relayed object 123H are projected into the angular range2720A and observed by observer 1050G. The occlusion plane 1510F isrelayed to relayed plane 150H. To summarize, the light rays relayed byrelay 5090B within display system 9002B and received by viewer 1050Hfill the FOV angular range 2720B, while the light rays relayed by relay5090A within display system 9002A and received by viewer 1050G fill theFOV angular range 2720A. The sum of these two angular ranges 2720A and2720B forms a combined FOV that is larger than the individual FOV of2720A or 2720B.

Each relay system 5090A and 5090B within the respective display system9002A and 9002B contains a relay comprised of two individualtransmissive reflector relays which may preserve the depth profile ofrelayed objects as discussed earlier. The one or more occlusion planes150F in relay system 9002B is closer to the relay formed by 5030F and5030G than the object 123F, and so it is relayed to relayed plane 150Hat a position further from the relay than the surface 123H relayed from123F. The separation between the occlusion plane 150F and the objectplane 123F may be set to be about equal to the distance between therelayed holographic object 121H and the relayed object 123H to provideocclusion of the background relayed object 123H for a foreground relayedholographic object 121H. For example, if light ray 133G reaches anobserver 1050H, then the observer 1050H can see a portion of thebackground relayed object 123H behind the relayed holographic object121H. The origin of light ray 133G is light ray 133K, which may beblocked by activating the occlusion region 151F, providing an observer1050H with an expected view of a foreground object 121H in front of abackground object 123H and blocking some of the light from thebackground object 123H. Similarly, for display system 9002A, if observer1050G can see light ray 1330G, which originates from object 1230F aslight ray 1330K, then the observer 1050G may perceive that the relayedholographic object 121H is transparent to the relayed background object123H. To avoid this, occlusion region 1510F may be activated to blocklight ray 1330K and prevent light 1330G from reaching observer 1050G.

More than two relays may be used in a relay system. FIG. 27G shows a toporthogonal view of a display system 2750 comprised of three individualrelays, each relaying light rays from an object D1-D3 into paths thatare divided into one of three angular ranges. FIG. 27H shows a sideorthogonal view of the same display system 2750 shown in FIG. 27G. Thenumbering from FIG. 27G is used in FIG. 27H. The light from any of theobjects D1-D3 may be combined with light from an image combining system10C, which will be discussed below. Object D1 2721A produces light alongpaths 2731A which reflect from mirror 2723A, and are directed toward atransmissive reflector 5030A, whereupon the light rays are relayed tolight rays 2731B, which converge at the relayed object 2725, andcontinue into the angular range 2726A. Similarly, light from object D32723A produces light along paths 2733A which reflect from mirror 2723Cand are received by transmissive reflector 5030C and then are relayedinto light paths 2733B which converge at the relayed object location2725 and continue into angular range 2726C. The side view in FIG. 27Hshows that light 2732A from object D2 2722A is reflected from an opticalfold mirror 2723B, received by a transmissive reflector relay 5030B, anddirected toward light paths 2732B, which contribute to forming relayedobject 2725 and continue on into angular range 2726B. The entire angularrange of light rays is the sum of the angular ranges 2726A, 2726B, and2726C. The plane 2724B is a possible occlusion plane, depending on thedetails of the image combining system 10C.

FIGS. 27I-L are orthogonal side views of several possibilities for theimage combining system 10C, which may be disposed in any of the paths oflight rays from D1-D3 2721A, 2722A, or 2723A. In each of the fourconfigurations shown in FIGS. 27I-L, input light paths 273X can be thelight paths 2731A from object D1 2721A, light paths 2732A from object D22722A, or light paths 2733A from object D3 2723A. FIG. 27I shows anorthogonal view of a light combining system with a light field displayand a relay. In FIG. 27I, light 2739A from a holographic object 2734Aprojected by a light field display 1001 is relayed by a relay 5030 intolight paths 2739B which form relayed holographic object 2734B, and thelight rays continue on to reflect from an image combiner 101 and areredirected to travel along with the paths of input light rays 273X. FIG.27J shows an orthogonal view of a light combining system with areal-world object and a relay system. In FIG. 27J, light paths 2741Afrom a real-world object 2740A pass through an occlusion plane 2724Abefore being received and relayed by relay 5030 into light paths 2741Bwhich converge to form the relayed image 2740B of real-world object2740A, these light paths 2741B reflecting from the image combiner andsent along with input light 273X. The occlusion plane 2724A may berelayed to relayed occlusion plane 2724B shown in FIGS. 27G and 27H andocclude portions of the real-world object as was discussed in referenceto FIG. 27F and earlier in this disclosure. FIG. 27K shows an orthogonalview of a light combining system with a real-world object. In FIG. 27K,light 2742 from a real-world object 2740A is redirected by the imagecombiner into light rays that travel with input rays 273X. FIG. 27Lshows an orthogonal view of a light combining system with a genericobject. In FIG. 27L, an object surface 2743 which may be a 2D displaysurface, a stereoscopic display surface, an autostereoscopic displaysurface, a multi-view display surface which may be the surface of ahorizontal parallax-only HPO multi-view display such as a lenticulardisplay, the surface or surfaces of a volumetric 3D display, a lightfield display surface, the surface of a real-world object emittinglight, or the surface of a real-world object reflecting light, or anyother type of object that reflects or emits light produces light 2744which is combined with the input light paths 273X by the image combiner101.

While in the example of FIGS. 27G and 27H there is almost no overlapillustrated between these three angular ranges 2726A, 2726B, and 2726C,some overlap is necessary to avoid dead regions of non-projectingdisplay area. The relay surface is defined by the three planes of thetransmissive reflectors 5030A, 5030B, and 5030C, and from differentviewpoints, there must not be seams visible to viewers 1050A, 1050B,1050C, or any other viewer in this combined FOV. FIG. 27M shows a frontview of the 3-sided relay system used in display system 2750 shown inFIGS. 27G and 27H, which may be viewed by a viewer 1050B in front ofdisplay system 2750 as shown in FIG. 27G. Light from locations 2735A and2736A reaches the viewer 1050B, and there is overlap between the panelsat these locations. However, as the viewer moves to the left and becomesviewer 1050A in FIG. 27G, the view may change. FIG. 27N shows a frontview of the 3-sided relay system used in display system 2750 shown inFIGS. 27G and 27H, which may be viewed by a viewer 1050A in front ofdisplay system 2750 as shown in FIG. 27G. Since only light relayed fromrelay surface 5030A reaches observer 1050A, the observer may not be ableto notice the gap 2735B between relay surfaces 5030B and 5030C. There isplenty of overlap on the seam near location 2736B between relay surfaces5030A and 5030B for this viewing position 1050A.

FIG. 27O is an orthogonal view of a display system comprising a relaysystem 2760 which relays light rays from an object that are projectedonly at wide angles relative to the surface of the relay system. Therelay system is comprised of two transmissive reflectors 5030A and5030B, where 5030A relays the light from an object 2751A to anintermediate relayed image 2751B. This light is received by transmissivereflector 5030B which relays the relayed image 2715B to a second relayedimage 2751C. The second relayed image 2751C is expected to havesubstantially the same depth profile as the source object 2751A. Thelight rays 2752A from object 2751A form a 45-degree incident angle withrespect to the normal to the surface of the first relay 5030A. Theselight rays, as well as the light rays from the object 2751A that liewithin a cone of about +/−22.5 degrees from these light rays 2752A willbe relayed into light rays grouped around light paths 2752B, forming therelayed object image 2751B. These light paths are within first andsecond ranges of angular alignment relative to the first transmissivereflector 5030A. These light rays 2752B are received by the second relay5030B, and are relayed into light paths 2752C, which may be seen by anobserver 1050C but not by observers 1050B or 1050A. Similarly, lightrays 2753A from object 2751A travel in a different direction from lightrays 2752A but also form a 45-degree incident angle with the normal ofthe surface of the first relay 5030A. These light rays 2753A, as well asthe light rays from the object 2751A that lie within a cone of about+/−22.5 degrees from these light rays 2753A will be relayed into lightrays grouped around light paths 2753B, also forming the relayed objectimage 2751B. These light rays 2753B are received by the second relay5030B, and are relayed into light paths 2753C, which may be seen by anobserver 1050A but not by observers 1050B or 1050C. Light paths from theobject 2751 that are along the path of normal incidence 2754 to thefirst relay 5030A surface, and most light rays within a cone of about+/−22.5 degrees away from this normal light path may be blocked by thetwo relays 5030A and 5030B, or may pass through relays 5030A and 5030Bwith some scattering. There may be one or more angle filters 2791 placedbetween the object 2751A and the first transmissive reflector 5030A toreject rays close to normal incidence to the relay surface 5030A so theydo not reach observer 1050B.

The relay system 2760 shown in FIG. 27O generates two fields of view forviewers 1050A and 1050C. It has some applications toward a table-topconfiguration, which will be discussed below. In an embodiment, a relaysystem comprises a first relay subsystem comprising: a transmissivereflector 5030A of the first relay subsystem, the first transmissivereflector positioned to receive image light from an image source 2751Aalong source light paths 2752A, 2753A within first and second ranges ofangular alignment relative to the transmissive reflector to form a firstimage surface, wherein the first transmissive reflector is configured torelay the image light to form a first relayed image surface 2751B in afirst relayed location; and a second transmissive reflector 5030B of thefirst relay subsystem, the second transmissive reflector positioned toreceive light from the first transmissive reflector and relay the lightfrom the first transmissive reflector to form a second relayed imagesurface 2751C in second relayed location; and wherein image light froman image source along source light paths outside of the first and secondrange of angular alignment relative to the transmissive reflector maynot be relayed to form a first image surface. In an embodiment, imagelight from the image source along the image source light paths that areoutside of the first and second ranges of angular alignment relative tothe first transmissive reflector are relayed by the first relaysubsystem with significantly more scattering than image light from theimage source along source light paths that are within the first andsecond ranges of angular alignment relative the first transmissivereflector. The first and second ranges of angular alignment relative tothe transmissive reflector comprise approximate ranges of −67.5 to −22.5degrees and +22.5 to +67.5 degrees relative to a normal to the surfaceof the transmissive reflector, respectively. In an embodiment, anoptional angle filter 2791 is employed between the image source 2751Aand the first transmissive reflector 5030A to absorb or reflect imagesource light along source light paths outside of the first and secondranges of angular alignment relative to the transmissive reflector. Inone embodiment the second relayed image surface is viewable in twodifferent viewing volumes with no overlap, and in another embodiment,there is overlap. The viewing volumes may be separated by 90 degreesfrom one another. In an embodiment, the second transmissive reflectormay form a table top, and the second relayed image surface is visible intwo viewing volumes substantially centered at −45 degrees and +45degrees relative to the normal of the table top, and viewable to twoviewers located on opposite sides of the table top.

FIG. 27P is an orthogonal side view of a display system 2770 comprisedof the display system shown in FIG. 27O with an added optical path forrelaying incident light paths that are close in angle to the normal ofthe surface of the first relay 5030A. FIG. 27Q is an orthogonal top viewof the display system 2770 shown in FIG. 27P. Some of the numbering fromrelay system 2760 shown in FIG. 27O is used in FIGS. 27P and 27Q. In anembodiment, the relay system of FIG. 27P is the relay system of FIG. 27Ofurther comprising: a first beam splitter 101A positioned to receive theimage light from the image source along the source light paths; a secondbeam splitter 101C and a second relay subsystem 5090 shown in FIG. 9J,wherein the first beam splitter 101A is configured to direct a firstportion of the image light from the image source 2751A to the firstrelay subsystem 5030A, 5030B and a second portion of the image lightfrom the image source to the second relay subsystem 5090; wherein thesecond relay subsystem 5090 is configured to relay light received fromthe first beam splitter 101A to the second beam splitter 101C; andwherein the second beam splitter is positioned to receive light from thesecond transmissive reflector 5030B of the first relay subsystem and isconfigured to combine the light from the second transmissive reflectorof the first relay subsystem 5030A, 5030B with light from the secondrelay subsystem 5090 and to direct the combined light to form the secondrelayed image surface 2751C. In an additional embodiment, the secondrelay subsystem comprises first and second transmissive reflectors5030C, 5030D of the second relay subsystem, wherein the firsttransmissive reflector 5030C of the second relay subsystem is positionedto receive light from the first beam splitter 101A and is configured torelay the received light to the second transmissive reflector 5030D ofthe second relay subsystem 5090, and wherein the second transmissivereflector 5030D of the second relay subsystem 5090 is configured torelay light from the first transmissive reflector 5030C of the secondrelay subsystem towards the second beam splitter 101C. In an additionalembodiment, the display system further comprises an additional imagesource 1001 operable to output additional image light along additionalsource light paths 2762A to form a second image surface 2756A, andwherein the second relay subsystem 5090 further comprises a first beamsplitter 101B of the second relay subsystem 5090 positioned to receiveand combine the additional image light from the additional image source1001 and the light 2754C from the first transmissive reflector of thesecond relay subsystem 5090 and direct the combined light to the secondtransmissive reflector 5030D of the second relay subsystem. In anembodiment, the relay system in 2770 further comprises an occlusionsystem operable to occlude a portion of light from the image source orthe additional image source. The occlusion system may comprise at leastone occlusion layer 2759A having one or more individually addressableelements or may comprise an occlusion object like 155A in FIG. 11C. Inone embodiment, light from the additional image source 1001 is relayedto an additional relayed image surface 2756B in proximity to the twicerelayed image surface 2751C, and wherein the occlusion system isoperable to occlude a portion of the light from the image source, theoccluded portion corresponding to a portion of the twice relayed imagesurface 2751C that is occluded by the additional relayed image surface2756B. In another embodiment, light from the additional image source2762A is relayed to an additional relayed image surface 2756B inproximity to the twice relayed image surface 2751C, and wherein theocclusion system is operable to occlude a portion of the light from theadditional image source, the occluded portion corresponding to a portionof the additional relayed image surface 2756B that is occluded by thetwice relayed image surface 2751C (not shown in FIGS. 27P and 27Q).

In both FIGS. 27P and 27Q, distance markers 2755 are used on the opticalaxes in the system to show one possible spacing between opticalcomponents, where the distance markers denote equivalent optical pathlength segments. In FIG. 27Q, the light from the object 2751A can becategorized into light rays 2752X centered around light path 2752A at a45-degree incidence to the first relay surface 5030A and with angularrange 2752Y, light rays 2753X centered around light path 2753A also at a45-degree incidence to the relay surface 5030A and with angular range2753Y, and light rays 2754X centered around a path 2754A normal to therelay surface 5030A which are not shown in detail in FIG. 27Q. In FIG.27Q, and as discussed for FIG. 27O, the light paths 2752X from object2751A centered around light path 2752A and found in angular range 2752Yare relayed to light rays 2752Z centered around light path 2752C, alsoin the same angular range 2752Y, forming a portion of relayed object2751C. Similarly, in FIG. 27Q, and as discussed for FIG. 27O, the lightpaths 2753X from object 2751A centered around light path 2753A and foundin angular range 2753Y are relayed to light rays 2753Z centered aroundlight path 2753C, also in the same angular range 2753Y, forming aportion of relayed object 2751C. Finally, as discussed in reference toFIG. 27O, the light paths 2754A from object 2751A centered around thenormal 2754 to the relay 5030A surface and found in the angular range2754Y are not relayed by the relay pair 5030A and 5030B. Instead, theserays are directed along a separate optical path through two separaterelays 5030C and 5030D as shown in FIG. 27P which is designed to passthis group of light rays that have close to normal incidence to therelay 5030A surface. These light paths 2754A are deflected by an imagecombiner 101A into light rays 2754B toward a third relay 5030C, which inthis instance is a transmissive reflector which receives the light paths2754B and relays these light paths 2754B to light paths 2754C which formthe first relayed object 2751B, continuing on be received by the secondrelay 5030B. An optional image combiner 101B may combine the relayedlight 2754C with the light 2762A from the surface of a holographicobject 2756A projected by a light field display 1001. In otherembodiments, image source 1001 may be a 2D display surface, astereoscopic display surface, an autostereoscopic display surface, amulti-view display surface which may be the surface of a horizontalparallax-only HPO multi-view display such as a lenticular display, thesurface or surfaces of a volumetric 3D display, a light field displaysurface, the surface of a real-world object emitting light, or thesurface of a real-world object reflecting light. The image combiner 101Bredirects the light 2756A from the holographic object into light rays2762B travelling substantially in the same direction as the light 2754Cfrom the relayed object 2751A. This combined light 2762B from theholographic object 2756A and the light 2754C from object 2751A isreceived by a fourth relay 5030D and relayed to combined light paths2762C and 2754D, respectively. An image combiner 101C redirects andcombines four sets of light paths: relayed light paths 2762C arereflected into light paths 2762D which converge to form relayedholographic object 2756B; light paths 2754D are reflected into lightpaths 2754E which converge to form the surface of relayed object 2751Cviewable by observer 1050B; light paths 2752Z grouped around 45-degreeangle light paths 2752C shown in FIG. 27Q which are relayed by relays5030A and 5030B and converge to form the surface of relayed object 2751Cviewable by observer 1050C; and light paths 2753Z grouped around45-degree angle light paths 2753C shown in FIG. 27Q which converge toform relayed object 2751C viewable by observer 1050A. All of these lightpaths exist in the group 2763 in FIG. 27P, but only the light that takesthe optical path through the relays 5030C and 5030D is shown in FIG.27P. The layer 2759A may be one or more occlusion planes which isrelayed to location 2759B, and may have individually-addressable regionsactivated so that the background relayed object 2751C may not be visiblebehind the relayed holographic object 2756B, much in the same way to theoperation of occlusion layers 151, 152, and 153 in FIG. 9A, and shown indetail in FIGS. 9B, 9C, and 9D. As explained with reference to FIG. 27O,there may be one or more angle filters placed between the object 2751Aand the first transmissive reflector 5030A to reject rays close tonormal incidence to the relay surface 5030A that pass through the imagecombiner 101A so they do not reach observer 1050B.

Display Systems with Multiple Separate Viewing Volumes

The relay in FIG. 27O relays light into two separate fields of viewdesigned for two observers viewing the display in two differentdirections. Such an application may be used in table-top displays, wherethe display surface is horizontal and the points of observation of thedisplay are above the display surface and may be on two or more sides ofthe display surface. FIG. 28A is an orthogonal view of a display systemin which the light rays from a holographic object 2801A projected by alight field display 1001 are split by a beam splitter into twodirections, with each direction providing a separate viewing volume. Inan embodiment, the image source 1001 may be a 2D display surface, astereoscopic display surface, an autostereoscopic display surface, amulti-view display surface which may be the surface of a horizontalparallax-only HPO multi-view display such as a lenticular display, thesurface or surfaces of a volumetric 3D display, a light field displaysurface, the surface of a real-world object emitting light, or thesurface of a real-world object reflecting light. Light rays 2802projected from a light field display 1001 form a holographic object2801A and are split by a beam splitter 101A into light rays 2803Apassing directly through the beam splitter 101A and 2804A deflected by101A, where light rays 2803A form a first viewing volume 2806A ofholographic object 2801A subtended by light rays 2803A, and light rays2804A form a second viewing volume 2805A of holographic object 2801Asubtended by light rays 2905A. Within the two dimensional view shown inFIG. 28A the first and second viewing volumes 2806A and 2805A,respectively, are shown as arcs subtending the group of light raysprojected from the corresponding holographic object, but it should beappreciated that each of these arcs indicates a viewing volume in space.Light rays 2804A appear to diverge from virtual holographic object2801B. Light rays 2803A and 2804A are received by a transmissivereflector relay 5030A, and are relayed into light paths 2803C and 2804C,forming relayed holographic objects 2801C and 2801D which may be viewedin viewing volumes 2805B and 2806B by observers 1050A and 1050B,respectively. The two angular ranges 2805B and 2806B that indicate theviewing volume for each relayed holographic object 2801C and 2801D,respectively, are not contiguous, as they are designed for two differentviewers. In an embodiment, the light field display 1001 in FIG. 28A isreplaced with a 2D display surface, a stereoscopic display surface, anautostereoscopic display surface, a multi-view display surface which maybe the surface of a horizontal parallax-only HPO multi-view display suchas a lenticular display, the surface or surfaces of a volumetric 3Ddisplay, the surface of a real-world object emitting light, or thesurface of a real-world object reflecting light. FIG. 28A is anembodiment of a relay system comprising at least one transmissivereflector 5030A; an image source 1001 operable to output light 2802, abeam splitter 101A positioned to receive the light from the image sourceand direct the light along first and second sets of source light paths2803A, 2804A wherein the image source and beam splitter are orientedrelative to the at least one transmissive reflector such that lightalong the first and second sets of source light paths is relayed alongfirst and second sets of relayed light paths 2803C, 2804C, respectively,the first and second sets of relayed light paths defining first andsecond relayed viewing volumes 2805A, 2806A, respectively; and whereinthe first and second relayed viewing volumes are different. In oneembodiment, first and second relayed viewing volumes partially overlap,while in another embodiment, first and second relayed viewing volumesdon't overlap. In an embodiment, the image source 1001 and beam splitter101A are oriented with respect with the at least one transmissivereflector 5030A such that the first and second sets of source lightpaths 2803A, 2804A respectively each comprise light paths orientedbetween 22.5 and 67.5 degrees relative to the surface of the at leastone transmissive reflector. In another embodiment, the image source 1001and beam splitter 101A are oriented with respect with the at least onetransmissive reflector 5030A such that the first and second sets ofrelayed light paths 2803C, 2804C respectively each comprise light pathsoriented between 22.5 and 67.5 degrees relative to the surface of the atleast one transmissive reflector 5030A.

FIG. 28B is an orthogonal view of a display system 2810 similar to FIG.28A, but with the light field display disposed out of the plane of therelay system, wherein the light from the light field display is directedtoward the relay system using an image combiner to allow light from anadditional source to enter the relay system. The numbering from FIG. 28Ais used in FIG. 28B. The light field display 1001 is disposed to projectlight 2802 along an optical axis which is substantially parallel to thesurface of the transmissive reflector relay 5030A. A side view 2810Afrom the viewpoint of observer 1050C shows that the light 2802 from thelight field display is split into two paths 2803A and 2804A by the beamsplitter 101A as was shown in FIG. 28A, but these light paths aredirected diagonally downward toward a beam splitter 101B. An end view2810B from the viewpoint of observer 1050D shows that all the light rays2834A from the light field display, comprised of both sets of light rays2803A and 2804A, are reflected by the beam splitter 101B into light rays2834B that are incident on the beam splitter 5030A, where light rays2834B comprise both light ray groups 2803B and 2804B. In this end view2810B, only light rays in one plane from the light field display 1001are shown. Groups of light rays 2803B and 2804B are received by therelay 5030A and relayed into groups of light rays 2803C and 2804C,respectively, forming the holographic objects 2801C and 2801D,respectively. The image combiner 101B is positioned to accept light 2811from another source separate from the light field display 1001.

The display system 2810 shown in FIG. 28B provides relayed holographicobjects in two separate fields of view above a relay surface, but it ispossible to use this system within a larger system to relay the lightfrom another object in addition to a holographic object, and alsoarrange for proper occlusion of a foreground holographic object with abackground object, or vice-versa. To accomplish this, the double-relayconfiguration 2760 shown in FIG. 27O is used. FIG. 28C is an orthogonaltop view of a display system which relays a background object withpossible occlusion along with a relayed holographic object by using thedisplay system 2810 shown in FIG. 28B and an additional relay system.The numbering of FIG. 28B is used in FIG. 28C. In FIG. 28C, the displaysystem 2810 shown in FIG. 28B is one stage of a two-stage relay systemwhich is comprised of the display system 2810 as the first stage, and atransmissive reflector 5030B as the second stage. The display system2810 receives light from an object 2811A, combines this light with thelight from holographic object 2801A, and relays this combined light toform both the relayed image 2811B of the object 2811A as well as therelayed holographic objects 2801C and 2801D. The details of relay 2810are discussed with reference to FIG. 28B. This relayed light fromdisplay system 2810 is received by the second-stage relay system 5030B,wherein once-relayed object image 2811B is relayed to twice-relayedobject image 2811C, and once-relayed holographic objects 2801C and 2801Dare relayed to twice-relayed holographic objects 2801E and 2801F,respectively. The light rays 2804C from once-relayed holographic object2801C subtend a holographic viewing volume 2805B, and these light raysare relayed by relay 5030B into light rays 2804D which form atwice-relayed holographic object 2801E viewable by observer 1050E in aviewing volume 2805C. The light rays 2803C from once-relayed holographicobject 2801D subtend a holographic viewing volume 2806B, and these lightrays are relayed by relay 5030B into light rays 2803D which form atwice-relayed holographic object 2801F viewable by observer 1050F in aviewing volume 2806C. In a similar manner, the occlusion plane 2851A isrelayed by display system 2810 to once-relayed occlusion plane 2851B,and this once-relayed occlusion plane 2811B is relayed by relay 5030B totwice-relayed occlusion plane 2851C. In FIG. 28C, the portion of relayedocclusion planes 2851B and 2851C which overlap with the respectiverelayed holographic objects 2801C/2801D and 2801E/2801F are not drawn.The depth ordering of the relayed object 2811C and the relayed occlusionplane 2851C is the same as the depth ordering of the object 2811A andthe occlusion plane 2851A, which allows the configuration of the displaysystem shown in FIG. 28C to handle occlusion properly. The occlusionplane 2851A may be offset from the object 2811A by a distance that issubstantially the same as the distance between the relayed holographicobjects 2801E and 2801F and the relayed object 2811C. Light from object2811A along paths 2813A, 2814A, and 2815A are relayed by display system2810 into light paths 2813B, 2814B, and 2815B, which are received byrelay 5030B and relayed into light paths 2813C, 2814C, and 2815C,respectively. The light paths 2813A and 2815A originate from the samepoint 2817A on the object 2811A, and their once-relayed light paths2813B and 2815B converge at the same corresponding point 2817B on therelayed object plane 2811B, while their twice-relayed light paths 2813Cand 2815C converge at the same corresponding point 2817C on the relayedobject plane 2811C. Observer 1050E can see light along path 2813C fromboth foreground relayed holographic object 2801E and background relayedobject 2811C simultaneously, which may not be desired. To avoid this andblock out light on or near light path 2813C, occlusion region 2888 onocclusion plane 2851A may be activated to a light-blocking state,preventing light on path 2813A from being relayed to light ray 2813C.Similarly, observer 1050F may be able to see light 2814C from backgroundrelayed object 2811C behind relayed holographic object 2801F. To blockout the background light 2814C, an occlusion site near location 2888 onocclusion plane 2851A may be activated to a light-blocking state. Thelight ray 2815C which helps form the relayed object 2811C should bevisible to observer 1050F, and so it's corresponding source ray 2815Ashould not be blocked by occlusion plane 2851A.

In an embodiment, a display system may further comprise an opticalcombiner 101B positioned to receive the light from the image source2803B, 2804B and receive light 2811 from an additional image source andconfigured to direct the combined light 2811 and 2803B, 2804B along thefirst and second sets of source light paths to the at least onetransmissive reflector 5030A, which is operable to relay the combinedlight from the first and second set of source light paths along thefirst and second set of relayed light paths 2811 and 2803C, 2804C intothe first 2805B and second 2806B viewing volumes, respectively. In anembodiment, the light from the image source and the additional imagesource are provided from different directions. In an embodiment, theadditional image source comprises any of: a 2D display surface, astereoscopic display surface, an autostereoscopic display surface, amulti-view display surface, the surface of a volumetric 3D display, alight field display surface, the surface of a real-world object emittinglight, or the surface of a real-world object reflecting light. In anembodiment, an input relay is configured to relay image light from theadditional image source to the optical combiner 101B (not shown in FIG.28B. The input relay is operable to relay image light from an additionalimage source to define a relayed image surface, whereby the additionalimage surface may comprise the relayed image surface of the additionalimage source; and wherein the optical combiner 101B is operable tocombine the light defining the relayed image surface of the additionalimage source with light from the image source and direct the combinedlight to the at least one transmissive reflector 5030A where thecombined light is relayed into the first and second viewing volumes. Inan embodiment, the display system further comprises an occlusion systemoperable to occlude a portion of light from at least one of the imagesource and the additional image source. The occlusion system maycomprise at least one occlusion layer having one or more individuallyaddressable elements, one or more occlusion objects, and be positionedto be optically preceding the optical combiner 101B.

In an embodiment, the light from the image source and the additionalimage source defines first 2801A, 2801B and second 2811A image surfaces,respectively, along the first and second sets of source light pathsrelayed by the transmissive reflector forming first relayed imagesurfaces 2801C, 2801D formed by first and second sets of relayed lightpaths 2804C, 2803C from the image source, respectively, and secondrelayed image surface 2811B formed by first and second sets of relayedlight paths 2813B and 2815B from the additional image source, andwherein the occlusion system 2851A is operable to occlude a portion ofthe light 2813A from the image source or the additional image source,the occluded portion 2813A corresponding to a portion of the first orsecond relayed image surface 2811B. In an embodiment, at least onetransmissive reflector comprises a first transmissive reflector 5030 andan additional transmissive reflector 5030B configured to relay lightalong the first and second sets of relayed light paths from the firsttransmissive reflector along third and fourth sets of relayed lightpaths 2804D, 2803D for the light from the image source, and third andfourth sets of relayed light paths 2813C and 2815D from the additionalimage source, wherein the light from the image source defines a sourceimage surface 2801A, 2801B along the first and second sets of sourcelight paths, the light relayed from the first transmissive reflectordefines a first relayed image surface 2801C, 2801D along the first andsecond sets of relayed light paths, and the light from the additionaltransmissive reflector defines a second relayed image surface 2801E,2801F along third and fourth sets of relayed light paths, wherein thefirst relayed image surface 2801C, 2801D has a first relayed depthprofile, and the second relayed image surface 2801E, 2801F has a secondrelayed depth profile that is different from the first relayed depthprofile but the same as a depth profile of the source image surface.

The display system shown in FIG. 28C may be used as a horizontal displaysurface surrounded by viewers that are located in a viewing range ofangles 2805C of twice relayed holographic object 2801E or located in theviewing range of angles 2806C of twice relayed holographic object 2801F.As described, these floating holographic objects 2801E and 2801F may beprojected in front of a relayed background object 2811C that is alsofloating, with proper occlusion handling for the portion of thebackground object 2811C that lies behind the relayed holographic objects2801E and 2801F as seen by one or more viewers in each of the twoholographic viewing volumes of the display system.

An alternate display system that may be used to project holographicobjects to one or more viewers in one or more holographic viewingvolumes is shown in FIG. 28D. FIG. 28D is an orthogonal view of adisplay system comprised of two or more holographic displays angled withrespect to the plane of a transmissive reflector relay. The light rays2843A projected from light field display 1001A form holographic object2844A viewable in a first holographic viewing 2847A, and these lightpaths 2843A are received by relay 5030C and relayed to relayed lightpaths 2843B forming relayed holographic object 2844B, viewable in athird holographic viewing volume 2847B by an observer 1050A. Similarly,the light rays 2841A projected from light field display 1001B formholographic object 2842A viewable in a second holographic viewing volume2846A, and these light rays 2841A are received by relay 5030C and arerelayed to relayed light paths 2841B which form relayed holographicobject 2842B viewable within a fourth holographic viewing volume 2846Aby an observer 1050B. Within the two dimensional view shown in FIG. 28Dthe first, second, third, and fourth viewing volumes 2847A, 2846A,2847B, and 2846B, respectively, are shown as arcs subtending the groupof light rays projected from the corresponding holographic object, butit should be appreciated that each of these arcs indicates a viewingvolume in space. The observers 1050A and 1050B may be seated across fromone another on opposite sides of a table with a top surface which iscomprised of relay 5030C, with the light field displays 1001A and 1001Bhidden from view beneath the table. In an embodiment, the display systemshown in FIG. 28D comprises a relay system comprising at least onetransmissive reflector 5030C, first and second image sources 1001A,1001B operable to output light along first and second sets of sourcelight paths 2843A, 2841A, respectively, wherein the first and secondimage sources 1001A, 1001B are oriented relative to the at least onetransmissive reflector such that light along the first and second setsof source light paths is relayed along first and second sets of relayedlight paths 2843B, 2841B, respectively, the first and second sets ofrelayed light paths defining first and second viewing volumes 2847B,2846B, respectively, wherein the first and second relayed viewingvolumes 2847B, 2846B are different. In an embodiment, the first andsecond relayed viewing volumes partially overlap, while in anotherembodiment, the first and second relayed viewing volumes do not overlap.In an embodiment, the first and second image sources 1001A, 1001B areoriented with respect with the at least one transmissive reflector 5030Csuch that the first and second sets of source light paths 2843A, 2841Aeach comprise light paths oriented between 22.5 and 67.5 degreesrelative to the at least one transmissive reflector 5030C. In anembodiment, the first and second image sources 1001A, 1001B are orientedwith respect to the at least one transmissive reflector 5030C such thatthe first and second sets of relayed light paths 2843B, 2841B eachcomprise light paths oriented between 22.5 and 67.5 degrees relative tothe at least one transmissive reflector 5030C. In another embodiment,the first and second image sources 1001A, 1001B each comprise a displaysurface oriented at an angle between 22.5 and 67.5 degrees relative tothe at least one transmissive reflector 5030C.

While the number of holographic displays in FIG. 28D is shown to be two,any number of light field displays may be disposed on one side of atransmissive reflector to create multiple relayed holographic objects atmultiple viewing locations. In one embodiment, any number of light fielddisplays may be arranged as shown in FIG. 28D on one side of atransmissive reflector, or on both sides of a transmissive reflector. Inanother embodiment, the light field displays are arranged so theindividual viewing volumes of one or more light field displays overlap.In another embodiment, the light field displays are arranged as shown inFIG. 28D, but in a substantially circumferential layout. In stillanother embodiment, the light field display sources 1001A and/or 1001Bdescribed in reference to FIG. 28D are replaced with any of: a 2Ddisplay surface, a stereoscopic display surface, an autostereoscopicdisplay surface, a multi-view display surface which may be the surfaceof a horizontal parallax-only HPO multi-view display such as alenticular display, the surface or surfaces of a volumetric 3D display,the surface of a real-world object emitting light, or the surface of areal-world object reflecting light.

FIG. 28E is a top view of an embodiment of the two-display system shownin FIG. 28D, wherein the display system comprises at least oneadditional image source. FIG. 28E is a table-top display systemcomprised of four displays arranged underneath a transmissive reflectorrelay with each display angled with respect to the plane of the relay sothat four holographic objects may be projected to viewers on each of thefour sides of the table. All the displays including displays 1001A and1001B in the display system 28E may be oriented in the same way asdisplays 1001A and 1001B in FIG. 28D, at roughly a 45 degree angle withthe surface of the transmissive reflector relay 5030C, as FIG. 28Ddemonstrates how projected light rays 2841A and 2843A formingholographic objects 2842A and 2844A are relayed by such an arrangement,respectively. Note that in FIG. 28D that a relayed holographic object2844B may be located directly over the projected holographic object2844A, so that from a top view of FIG. 28D, these objects 2844B, 2844Aare coincident. The same is true for the holographic objects and relayedholographic objects shown in FIG. 28E. In FIG. 28E, light rays 2886Aprojected by display 1001A form the first projected holographic surface2880A, and these light rays diverge until they are received and relayedinto light rays 2886B by the relay 5030C to form first relayedholographic object surface 2880E, viewable within a first viewing volume2891 subtended by light rays 2886B by observer 1050A. Holographic object2880A and relayed object 2880E are coincident in the top view of FIG.28E. The light rays 2886A forming holographic surface 2880A travel underthe tabletop relay 5030C, denoted as dashed lines, and the relayed lightrays 2886B forming relayed holographic object surface 2880E travel overthe tabletop, denoted as solid lines. In a similar way, light rays 2887Aprojected by display 1001B and forming second projected holographicobject surface 2880B underneath the tabletop are relayed by relay 5030Cinto relayed light rays 2887B forming second relayed holographic objectsurface 2880F observed by viewer 1050B in the second viewing volume 2892subtended by light rays 2887B. Light rays 2888A projected by display1001C and forming third projected holographic object surface 2880Cunderneath the tabletop are relayed by relay 5030C into relayed lightrays 2888B forming third relayed holographic object surface 2880Fobserved by viewer 1050C in the third viewing volume 2893 subtended bylight rays 2888B. And finally, light rays 2889A projected by display1001D and forming fourth projected holographic object surface 2880Dunderneath the tabletop are relayed by transmissive reflector relay5030C into relayed light rays 2889B forming fourth relayed holographicobject surface 2880H observed by viewer 1050D in the fourth viewingvolume 2894 subtended by light rays 2889B. The relayed holographicsurfaces 2880E-H may be the same or different, as in FIG. 28E thesurfaces 2880E and 2880G may be the same, but different from surfaces2880F and 2880H. In the display system shown in FIG. 28E, there are fourdisplays used to create four non-overlapping viewing volumes forholographic objects that are each independent. In other embodiments,other configurations include more sides to the table top 5030C, more orfewer displays than four, and more or less than four viewing volumes,where some of the viewing volumes corresponding to one or more displaysmay or may not overlap. In still another embodiment, one or more of thelight field displays 1001A-D described in reference to FIG. 28E arereplaced with are replaced with any of: a 2D display surface, astereoscopic display surface, an autostereoscopic display surface, amulti-view display surface which may be the surface of a horizontalparallax-only HPO multi-view display such as a lenticular display, thesurface or surfaces of a volumetric 3D display, the surface of areal-world object emitting light, or the surface of a real-world objectreflecting light. In an embodiment shown in FIG. 28E, the display systemof FIG. 28D comprises at least one additional image source 1001C, 1001Doperable to output light along at least one additional set of sourcelight paths 2888A, 2889A wherein the at least one additional imagesource is oriented relative to the at least one transmissive reflector5030C such that light along the at least one additional set of sourcelight paths 2888A, 2889A are relayed along at least one additional setof relayed light paths 2888B, 2889B, respectively, the at least oneadditional set of relayed light paths defining at least one additionalviewing volume 2893, 2894, and wherein the at least one additionalrelayed viewing volume 2893, 2894 is different from any other viewingvolumes 2891, 2892.

FIG. 28F is an orthogonal view of a display system comprised of two ormore image combining systems angled relative to the surface of atransmissive reflector relay, each image combining system combininglight from a holographic object and another object, with the combinedlight from each image combining system relayed to a separate location,the separate locations designed for viewing by separate viewers. Thefirst image combining system is comprised of light field display 1001E,object 2852A, transmissive reflector relay 5030A, and image combiner101A. Light rays 2861A projected from light field display 1001E formholographic object 2842A and pass through an image combiner 101A. Lightrays 2853A from an object 2852A pass through one or more occlusionplanes 2854A and are relayed into light paths 2853B by transmissivereflector relay 5030A, forming first relayed object 2852B. The lightpaths 2853B are reflected by the image combiner 101A into light paths2853C which are combined with the light rays 2861A from the holographicobject 2842A. These combined light paths 2853C and 2861A are received bythe relay 5030C and relayed into light paths 2853D and 2861B,respectively, wherein light paths 2853D converge to form relayed object2852C and light paths 2861B converge to form relayed holographic object2842B. The occlusion plane 2854A near object 2852A is relayed to relayedocclusion plane 2854C. The portion of the relayed occlusion plane 2854Awhich overlaps with the relayed holographic object 2842B is not shown inFIG. 28F.

An observer 1050A may observe relayed holographic object 2842B, but notsee light rays from the relayed object 2852C directly behind theholographic object 2842B if the center rays in the group of rays 2853Dare missing. This occlusion may be achieved by occluding the centerportion of corresponding light rays 2853A from the object 2852A byactivating the occlusion plane locations 2855 on occlusion plane 2854Ato block light. The distance between the occlusion plane 2854A and theobject 2852A may be substantially the same as the distance between therelayed holographic object 2842B and the relayed object 2852C. Thedouble relay of light from object 2852A through transmissive reflector5030A followed by transmissive reflector 5030C substantially preservesthe depth of the object 2852A for the corresponding relayed object2852C, as well as maintaining the depth ordering of the one or moreocclusion planes 2854A in front of the object 2852A so that thecorresponding relayed occlusion planes 2854C may be placed insubstantially the same location as the relayed holographic object 2842B.In an embodiment, the display system in FIG. 28F comprises a firstoptical combiner 101A positioned to receive the light 2861A from thefirst image source 1001E and light from a third image source 2852A andconfigured to direct combined light 2861A, 2853C to the at least onetransmissive reflector 5030C, which is operable to relay the combinedlight into the first viewing volume 2896A. In an embodiment, the thirdimage source comprises any of: a 2D display surface, a stereoscopicdisplay surface, an autostereoscopic display surface, a multi-viewdisplay surface, the surface of a volumetric 3D display, a second lightfield display surface, the surface of a real-world object emittinglight, or the surface of a real-world object reflecting light. In anembodiment, the display system comprises an input relay 5030A, whereinthe input relay 5030A is configured to relay image light from thirdimage source to the first optical combiner 101A. The input relay 5030Ais operable to relay image light from a surface of the third imagesource 2852A to define a first relayed image surface 2852B, whereby thethird image surface comprises the first relayed image surface 2852B, andwherein the first optical combiner 101A is operable to combine the lightdefining the first relayed image surface 2852B with light from the firstimage source 2853A and direct the combined light to the at least onetransmissive reflector 5030C where the combined light is relayed intothe first viewing volume 2896A. In an embodiment, the combined light2861B, 2853D relayed from the at least one transmissive reflectordefines at least a second relayed image surface 2852C of the third imagesource 2852A in the first viewing volume 2896A, and wherein the firstrelayed image surface 2852B has a first relayed depth profile, and thesecond relayed image surface 2852C has a second relayed depth profilethat is different from the first relayed depth profile 2852B but thesame as a depth profile of the surface of the third image source 2852A.

The second image combining system in FIG. 28F is comprised of lightfield display 1001F, object 2862A, transmissive reflector relay 5030B,and image combiner 101B. Light rays 2871A projected from light fielddisplay 1001F forming holographic object 2844A pass through an imagecombiner 101B. Light rays 2863A from an object 2862A pass through one ormore occlusion planes 2864A and are relayed into light paths 2863B bytransmissive reflector relay 5030B, forming first relayed object 2862B.The light paths 2863B are reflected by the image combiner 101B intolight paths 2863C which are combined with the light rays 2871A from theholographic object 2844A. These combined light paths 2863C and 2871A arereceived by the relay 5030C and relayed into light paths 2863D and2871B, respectively, wherein light paths 2863D converge to form relayedobject 2862C and light paths 2871B converge to form relayed holographicobject 2844B. In an embodiment, the display system further comprises asecond optical combiner 101B positioned to receive the light 2871A fromthe second image source and light 2863A from a fourth image source 2862Aand configured to direct combined light from the second optical combiner101B to the at least one transmissive reflector 5030C, which is operableto relay the combined light of the second optical combiner 101B into thesecond viewing volume 2896B. In an embodiment, the fourth image sourcecomprises any of: a 2D display surface, a stereoscopic display surface,an autostereoscopic display surface, a multi-view display surface, thesurface of a volumetric 3D display, a light field display surface, thesurface of a real-world object emitting light, or the surface of areal-world object reflecting light. In an embodiment, the display systemof FIG. 28F comprises an input relay 5030B, wherein the input relay isconfigured to relay image light 2863A to the second optical combiner101B. In an embodiment, the input relay 5030B is operable to relay imagelight from a fourth image source 2862A to define a first relayed imagesurface, whereby the fourth image surface comprises the first relayedimage surface 2862B; and wherein the second optical combiner 101B isoperable to combine the light 2863B defining the first relayed imagesurface 2862B with light 2871A from the second image source 1001F anddirect the combined light to the at least one transmissive reflector5030C where the combined light is relayed into the second viewingvolume. In an embodiment, the combined light from the at least onetransmissive reflector 5030C defines at least a second relayed imagesurface 2862C of the fourth image source in the second viewing volume2896B and wherein the first relayed image surface 2862B of the fourthimage surface has a first relayed depth profile, and the second relayedimage surface 2862C of the fourth image surface has a second relayeddepth profile that is different from the first relayed depth profile of2862B but the same as a depth profile of the surface of the object2862A.

In an embodiment, the display system of FIG. 28F further comprises anocclusion system operable to occlude a portion of light from at leastone of the first 1001E and third 2852A image sources. In one embodiment,the occlusion system comprises at least one occlusion layer 2854A havingone or more individually addressable elements 2855. In anotherembodiment, the occlusion system comprises at least one occlusion object(not shown). The occlusion system may be positioned to be opticallypreceding the optical combiner 101A. In an embodiment, the light fromthe first 1001E and third 2852A image sources defines first 2842A andsecond 2852B image surfaces, respectively, and this light is relayed bythe at least one transmissive reflector 5030C to define first 2842B andsecond 2852C relayed image surfaces in the first viewing volume 2896A,and wherein the occlusion system 2854A is operable to occlude a portionof the light from the first or third image source 2852B, the occludedportion corresponding to a portion of the first or second relayed imagesurface 2852C that is occluded by the other one of the first or secondimage relayed image surface 2842B viewed by 1050A.

In an embodiment, the display system of FIG. 28F further comprises anocclusion system operable to occlude a portion of light from at leastone of the second 1001F and fourth 2862A image sources. In oneembodiment, the occlusion system comprises at least one occlusion layer2864A having one or more individually addressable elements 2865. Inanother embodiment, the occlusion system comprises at least oneocclusion object (not shown). The occlusion system may be positioned tobe optically preceding the optical combiner 101B. In an embodiment, thelight from the first 1001F and fourth 2862A image sources defines first2844A and second 2862B image surfaces, respectively, and this light isrelayed by the at least one transmissive reflector 5030C to define first2844B and second 2862C relayed image surfaces in the second viewingvolume 2896B, and wherein the occlusion system 2864A is operable toocclude a portion of the light from the first or fourth image source2862A, the occluded portion corresponding to a portion of the first orsecond relayed image surface 2862C that is occluded by the other one ofthe first or second relayed image surface 2844B and viewed by viewer1050B.

The occlusion plane 2864A near object 2862A is relayed to relayedocclusion plane 2864C. The portion of the relayed occlusion plane 2864Cwhich overlaps with the relayed holographic object 2844B is not shown inFIG. 28F. An observer 1050B may observe relayed holographic object2844B, but not see light rays from the relayed object 2862C directlybehind the holographic object 2862B if the center rays in the group ofrays 2863D are missing. This occlusion may be achieved by occluding thecenter portion of corresponding light rays 2863A from the object 2862Aat the occlusion plane location 2865. The distance between the occlusionplane 2864A and the object 2862A may be substantially the same as thedistance between the relayed holographic object 2844B and the relayedobject 2862C. The double relay of light from object 2862A throughtransmissive reflector 5030B followed by transmissive reflector 5030Csubstantially preserves the depth profile of the object 2862A for thecorresponding relayed object 2862C, as well as maintaining the depthordering of the one or more occlusion planes 2864A in front of theobject 2862A so that the corresponding relayed one or more occlusionplanes 2864C may be placed in substantially the same location as therelayed holographic object 2844B.

Many display variations of the display system shown in FIG. 28F arepossible. In an embodiment, the holographic displays 1001E and 1001F aswell as the objects 2852A and 2862A in FIG. 28F can be any of: a 2Ddisplay surface, a stereoscopic display surface, an autostereoscopicdisplay surface, a multi-view display surface which may be the surfaceof a horizontal parallax-only HPO multi-view display such as alenticular display, the surface or surfaces of a volumetric 3D display,the surface of a real-world object emitting light, or the surface of areal-world object reflecting light.

Modular Display Systems

FIG. 29A shows a top view of two display devices 201, one for placementon first imaging plane A, and the other for placement on a secondimaging plane B, each display device comprised of a display area 205 anda non-imaging area 206, which may be a bezel, for example. FIG. 29Bshows a side view and an end view of the display device 201. The displaydevices 201 may be emissive displays such as LED, OLED, or micro-LEDdisplays, or transmissive displays such as an LCD display. FIG. 29Cshows multiple displays 201 placed on a first plane A 211, and multipledisplays 201 placed on a second plane B 212. FIG. 29D shows a side viewof first display plane A 211 and second display plane B 212 disposedorthogonal to one another, the light 241 from plane A 211 superimposedtogether with the light 242 from plane B 212 to form superimposed light243 using a light combining system comprising an optical image combiner101, where the superimposed light 243 reaches an observer 1050. Theoptical combiner 101 may be a non-polarizing beam splitter, a polarizingbeam splitter, a half-mirror, or some other optical system, which maycontain refractive optics, diffractive optics, or mirrored systems. FIG.29E shows the combined light 243 as viewed by the observer 1050, withdisplay plane A 211 and display plane B 212 superimposed, with thedisplays 201 on plane B shown with dashed lines and slightly faded todistinguish them from displays 201 on plane A. The small shift betweenthe planes allows the formation of regions 221, where a non-imagingregion producing no light from display plane B 212 is overlapped with animaging region on display plane A 211 producing light, so that somelight may be produced in this region 221 from at least one display.There are still regions 222 wherein the non-imaging areas from displayson the planes overlap, and these regions produce no light. If thenon-imaging areas are negligible in size, then this overlap region maybe acceptable, but for practical displays this non-imaging region isusually substantial enough to be noticed by an observer 1050.

Other arrangements of display planes may be superimposed using beamsplitter configurations similar to the one shown in FIG. 29D. FIG. 29Fshows two display planes of display devices 201 placed on a regularrectangular grid, display plane D 214 and display plane E 215, offsetfrom one another in two dimensions by a small amount in order tomaximize the overlapping regions 217 where at least one display planeproduces light, and minimize the non-imaging regions 218 overlap on bothdisplay planes in which neither display plane D 214 nor E 215 produceslight.

It is possible to use display planes that are rotated with respect toone another. FIG. 29G shows two overlapped display planes A 211 and B212 shown separately in FIG. 29C, where the display plane A 211 isrotated 90 degrees relative to the other display plane B 212. As inprevious configurations, this causes regions 221 where there is one butonly one non-imaging region on one of the planes, and non-imagingregions 222 where there are non-imaging regions on both planes A 211 andB 212. It is possible to use a third display plane with non-imagingregions to eliminate these non-imaging regions 222. FIG. 29H shows adisplay plane C 213 comprised of a regular rectilinear grid of displaydevices 201 placed size-by-side in neat rows. FIG. 29I shows a side viewof one embodiment of a display system 2910 comprised of a lightcombining system comprising at least two optical combiners 101A and 101Bcombining the light from three display planes A 211, B 212, and C 213.Display planes B 212 and C 213 are placed parallel with respect to oneanother but may be offset so that the distance between display plane B212 and beam splitter 101B is the same distance between display plane A211 and the beam splitter 101B. Plane B 212 is rotated 90 degreesrelative to plane C 213 so that for the side view shown in FIG. 29I, thetwo long sides of displays 201B are visible on display plane B 212,while the three short sides of displays 201C are visible on displayplane C 213. Display plane A 211 is disposed orthogonally to displayplane C 213, and for the side view shown in FIG. 29I the short sides ofdisplay devices 201A in plane A 211 are visible. Light 241 from thesurface 280A of display plane A 211 may be combined with the light 251from the surface 280C of display plane C 213 into combined light 252.This combined light 252 is combined with the light 242 from the surface280B of display plane B 212 into combined light 253 from the threedisplay planes A 211, B 212, and C 213, which reaches observer 1050.Observer 1050 sees the combined light 274 as if it came from a singledisplay that is at the distance between the observer 1050 and displayplane A 211. The optical path length between the observer 1050 and anyof the three display planes A 211, B 212, or C 213 may be adjusted to besubstantially the same. These equal path lengths may be necessary if thecombined light 253 is to be relayed so that it is focused at a virtualdisplay plane.

FIG. 29J is the combined light 253 observed by observer 1050 from thethree display planes shown in FIG. 29I. Display planes A 211 and C 213are parallel but offset from each other by less than a short dimensionof the display device. Display plane B 212 is orthogonal to displayplanes A 211 and C 213. The display planes have been aligned so thatthere may be locations 219 with only one bezel from one display plane atthe corresponding location, but display regions existing on the othertwo planes, or locations 220 with a display regions existing on onedisplay plane (e.g. display plane C 213), but perhaps only one due totwo non-imaging regions on the other two planes (e.g. planes A 211 and B212) at the location 220. In FIG. 29J, every location has at least onedisplay source on one of the three display planes. In this way, thecombined light from display planes A 211, B 212, and C 213 shown in FIG.29J is a seamless display surface 280, which has a combined resolutionof many separate display devices 201, where each separate display device201 contains a non-imaging region. The seamless display surface 280shown in FIG. 29J composed of three contributing planes of displaysurfaces 280A, 280B, and 280C shown in FIG. 29I may be made as large andwith as high a resolution as desired, provided that optical combiners101A and 101B in FIG. 29I may be made suitably large.

One possible advantage of placing display devices on display planeswhich are rotated with respect to one another (e.g. display planes A 211and B 212 in FIG. 29J) is an increase in resolution of the combinedpixels that result from the overlap of pixels on more than one displayplane. For example, in some embodiments, the display pixels on eachplane will be comprised of more than one subpixel. FIG. 29K shows anembodiment in which each pixel such as 230 or 235 is comprised of threerectangular subpixels, which may be red, blue, and green in color.Corresponding to the display arrangement shown in FIG. 29J, it ispossible that the subpixels 231, 232, and 233 (e.g. red, green, and bluesubpixels) that form A-plane pixel 230 for the displays on plane A 211may be taller than they are wider, which means that the subpixels 236,237, and 238 (e.g. red, green, and blue subpixels) that form B-planepixel 235 for the displays on plane B 212, rotated to be orthogonal tothe display devices on plane B, may be wider than they are taller. Afterbeing superimposed, pixel 230 and pixel 235 may result in the crossedsuperimposed subpixel pattern 240, containing 9 crossed subpixel regionssuch as 234. The larger number of crossed subpixel regions onsuperimposed pixel 240 may offer more color choices and a highereffective spatial resolution than the combined number of sourcesubpixels from pixels 230 and 235.

While the seamless display surface 280 shown in FIG. 29J may not havenon-display regions, it is composed of three contributing planes ofdisplays A 211, B 212, and C 213, each of which has displays placed in aclose-packed formation. Other more efficient arrangements of displaydevice planes are possible. FIG. 29L shows four identical displayplanes, display plane I 216, display plane J217, display plane K 218,and display plane L 219, each comprised of a pattern of displays 201with spaces between each display 201 and its neighbors. While thesedisplay planes only show four displays each, they may be made as largeas desired with the same display-to-display separation in each axis.These four display planes may be combined using a light combining systemwith one or more optical combiners much the same way that three displayplanes are combined in FIG. 29I. FIG. 29M shows how four display planesI 216, J 217, K 218, and L 219 shown in FIG. 29L may be combined usingthree optical combiners 101A, 101B, and 101C of a light combining systemto form an overlapped 2D display system 2920. The light 261 from displayplane I 216 surface 2901 and the light 262 from display plane J 217surface 290J are combined by beam splitter 101A into combined light I+J263. The light 271 from display plane K 218 surface 290K and the light272 from display plane L 219 surface 290L is combined by beam splitter101B into combined light K+L 273. The light I+J 263 and the light K+L273 is combined by beam splitter 101C into the combined light 274I+J+K+L seen by observer 1050. FIG. 29N shows that observer 1050 shouldsee overlapping display planes 275 from the configuration shown in FIG.29M, with an effective overlapped seamless 2D display surface 290. FIG.29O shows the configuration of four overlapping display planes I 216, J217, K 218, and L 219 that produce the combined light 274 I+J+K+L seenby observer 1050 from the configuration shown in FIG. 29M. These fouroverlapping display planes I 216, J 217, K 218, and L 219 havenon-imaging regions overlap in some regions 265 where at most three, butnever four non-imaging regions overlap simultaneously. This means thatsubstantially all regions on combined seamless display surface 290produce light. The seamless display surface 290 shown in FIG. 29Ocomprised of four contributing planes of displays may be made as largeas desired, and with as high a resolution as desired, provided thatoptical combiners 101A, 101B, and 101C may be made suitably large. Theconfigurations shown in this disclosure are exemplary, and many otherconfigurations of display planes with non-imaging area may be combinedto produce one effective seamless display plane.

In view of the principles illustrated with the above examples, it is tobe appreciated that, generally, a display system can be constructed toinclude arrays of modular display devices, each modular display devicecomprising a display area and a non-imaging area, wherein the arrays ofmodular display devices define a plurality of display planes, eachdisplay plane comprising imaging regions defined by the display areas ofthe respective display devices and non-imaging regions defined by thenon-imaging areas of the respective display devices. Further, thedisplay system can be constructed to further include a light combiningsystem operable to combine light from the arrays of modular displaydevices, wherein the light combining system and the arrays of modulardisplay devices are arranged such that the combined light has aneffective display plane defined by superimposing the plurality ofdisplay planes so that the non-imaging regions of the plurality ofdisplay planes are superimposed by the imaging regions of the pluralityof display planes.

Seamless display planes with resolution that may be made as large asrequired may be combined with arrays of waveguides in order to createlight field display systems. FIG. 30A shows a single waveguide 1004Aplaced over an illumination plane 3002 which is comprised ofindividually addressable pixels 3003 at coordinates u₀ 3010, u_(k) 3011,and u_(−k) 3012 located on a seamless display surface 3020. The seamlessdisplay surface 3020 may be seamless display surface 290 in FIG. 29O,seamless display surface 280 in FIG. 29J, the display area 205 ofdisplay device 201 shown in FIG. 29A, or some other display surface. Theillumination plane 3002 may be an embodiment of the display area 205from display device 201 shown in FIGS. 29A and 29B. The illuminationplane 3002 contains pixels in a plane defined by two orthogonal axes U3005 and V 3006, but in FIG. 30A pixels 3002 are only shown in theU-axis 3005. Each waveguide is associated with a group of pixels 3002. Awaveguide 1004A will receive light 3041 from pixel u_(−k) 3012 on theillumination plane 3002 and project this light 3041 into a direction3031 defined by an angle determined at least in part by the location ofthe pixel 3012 on the U-V plane with respect to the waveguide 1004A.Some of the light 3042 from the pixel at the left u_(k) 3011 is receivedby the waveguide 1004A and propagated into chief ray propagation path3032, the direction of 3032 up and to the right determined by thelocation of pixel u_(k) 3011 relative to the waveguide 1004A. The chiefray propagation path 3030 that is normal to the illumination plane isprovided in this example by the light from pixel u₀ 3010 close to theoptical axis of the waveguide 1004A. The coordinates u₀, u_(k), andu_(−k) are light field angular coordinates of light propagation paths inone dimension, called axis U, but there is a corresponding angularcoordinate in the orthogonal dimension V. In general, the waveguide1004A is assigned to have a single spatial coordinate in two dimensions(X, Y), and a pixel 3003, 3010, 3011, or 3012 associated with awaveguide may produce a light propagation path with a two-dimensionalangular coordinate (U, V). Together, these 2D spatial coordinates (X, Y)and 2D angular coordinates (U, V) form a 4-dimensional (4D) light fieldcoordinate (X, Y, U, V) assigned to each pixel 3003, 3010, 3011, or 3012on the illumination plane 3002.

The 4D light field is comprised of all the 4D coordinates (X, Y, U, V)for multiple waveguides at various spatial coordinates, each waveguide1004A associated with multiple angular coordinates (U, V) correspondingto the illumination source pixels 3003 associated with the waveguide1004A (e.g. spanning u_(−k), and u_(k) in the U-axis 3005 for waveguide1004A shown in FIG. 30A). FIG. 30B shows a light field display system3060 comprised of a plane of waveguides 1004 disposed over anillumination plane 3002 which contains illumination sources (e.g.pixels) 3003 and forms a seamless display surface 3020. The seamlessdisplay surface 3020 may be seamless display surface 290 in FIG. 29O,seamless display surface 280 in FIG. 29J, the display area 205 ofdisplay device 201 shown in FIG. 29A, or some other display surface.Above the illumination plane is a waveguide array 1004 comprised ofthree waveguides 1004A, 1004B, and 1004C. Associated with each waveguide1004A, 1004B, and 1004C is a group of pixels 3002A, 3002B, and 3003B,which produce groups of propagation paths 3025A, 3025B, and 3025C,respectively. The chief rays 3031, 3030, and 3032 define the propagationpaths of light projected from the waveguide 1004A at the minimum,mid-value, and maximum values of light field angular coordinate U,respectively. The light field angular coordinate V is orthogonal to U.In FIG. 30B, the light-inhibiting structures 3009 forming vertical wallsbetween neighboring waveguides 1004A, 1004B, and 1004C prevent lightgenerated by one group of pixels associated with a first waveguide fromreaching the neighboring waveguide. For example, light from any pixel3002B associated with the center waveguide 1004B cannot reach waveguide1004A because of the light-inhibiting structure 3009 between these twowaveguides.

FIG. 30C shows a side view of a light field display 3050 comprised ofthe display device 201 shown in FIG. 29B with a waveguide array such as1004 shown in FIG. 30B mounted an active display area. This light fielddisplay projects light rays into propagation paths as shown in FIG. 30B.Below, this disclosure demonstrates how this building block 3050 may beused as a building block in a light field display with a higherresolution than the light field display 3050.

FIG. 30D shows a display device 201 with an active display area 205covered with an array of waveguides 1004, surrounded by a non-imagingarea 206. A magnified view 3030 of the two waveguides 1004A at(X,Y)=(0,0) and 1004B at (X,Y)=(1,0) shows the U,V, and Z-axes 3040 thatare also shown in FIG. 30A, as well as the 4-D pixel coordinatesassociated with each waveguide. These pixels collectively form anillumination source plane 3002 which is also shown in FIG. 30B. Forexample, pixel 3083 is associated with (X,Y,U,V) coordinates(0,0,−2,−2), denoted by x₀y₀u⁻²v⁻². The pixel 3093, under the samerelative location relative to waveguide 1004B as the location of pixel3083 relative to waveguide 1004A, has the same (U,V) coordinate (−2,−2),with (X,Y,U,V) coordinate (1,0,−2,−2). Similarly, pixel 3081 at thecenter of waveguide 1004A, has (X,Y,U,V) coordinate (0,0,0,0), whilepixel 3091 at the center of waveguide 1004B, has (X,Y,U,V) coordinate(1,0,0,0). Some other 4D light field coordinates are shown in FIG. 30D,including (X,Y,U,V)=(0,0,−1,0), (0,0,−2,0), (0,0,−3,0), and (1,0,0,−1).

FIG. 30E shows two holographic objects 3022 and 3024 projected by alight field display system comprised of five waveguides 1004A-E, eachprojecting light from a group of associated pixels 3002A-D,respectively, and perceived by an observer 1050. The pixels are part ofa seamless display surface 3020, which may be seamless display surface290 in FIG. 29O, seamless display surface 280 in FIG. 29J, the displayarea 205 of display device 201 shown in FIG. 29A, or some other displaysurface. The light rays defined by chief rays 3023 forming holographicobject 3024 include light from pixel 3071 projected by waveguide 1004A,light from pixel 3072 projected by waveguide 1004B, and light from pixel3073 projected by waveguide 1004C. The light rays defined by chief rays3021 forming holographic object 3022 include light from pixel 3074projected by waveguide 1004C, light from pixel 3075 projected bywaveguide 1004D, and light from pixel 3076 projected by waveguide 1004E.In FIG. 30E, the light-inhibiting structures 3009 forming vertical wallsbetween neighboring waveguides 1004A-D prevent light generated by onegroup of pixels associated with a first waveguide from reaching aneighboring waveguide. For example, light from any pixel 3002Cassociated with the waveguide 1004C cannot reach waveguide 1004B orwaveguide 1004D because the light-inhibiting structures 3009 surroundingwaveguide 1004C would block and absorb this stray light. While only thegroups of chief ray propagation paths 3023 and 3021 are shown in FIG.30E, it should be appreciated that the light from the illuminationsource plane pixels 3071-3076 may substantially fill the apertures ofthe respective waveguides, just as the light 3041 from pixel 3012substantially fills the aperture of waveguide 1004A as this light 3041is projected into chief ray propagation path 3031 in FIG. 30A.

It should be noted that throughout this disclosure, any light fielddisplay may be converted to a normal display with the addition of alayer of switchable glass (e.g. “smart glass”), which is layer of glassor glazing with light transmission properties that change fromtransparent to translucent when voltage, light, or heat is applied. Forexample, in polymer-dispersed liquid-crystal devices (PDLCs), liquidcrystals are dissolved or dispersed into a liquid polymer followed bysolidification or curing of the polymer. Typically, the liquid mix ofpolymer and liquid crystals is placed between two layers of transparentand conductive glass or plastic followed by curing of the polymer,thereby forming the basic sandwich structure of the smart window.Electrodes from a power supply are attached to the transparentelectrodes. With no applied voltage, the liquid crystals are randomlyarranged in the droplets, resulting in scattering of light as it passesthrough the smart window assembly. This results in a translucent, milkywhite appearance. When a voltage is applied to the electrodes, theelectric field formed between the two transparent electrodes on theglass causes the liquid crystals to align, allowing light to passthrough the droplets with very little scattering and resulting in astate with varying transparency depending on the voltage applied.

FIG. 30F shows the light field display 3060 shown in FIG. 30B, with alayer of smart glass 3070 placed in a plane parallel to the plane ofwaveguides 1004 and displaced a small distance from the surface of thewaveguides 1004. The numbering of FIG. 30B is used in FIG. 30F. Asubstrate 3071, which may be a mix of cured polymer and liquid crystals,with the liquid crystal molecules forming droplets in the polymer, liesbetween two transparent plastic or glass electrode plates 3072. Avoltage source 3075 is attached to the electrode plates 3072 and appliesa voltage to the substrate 3071 between the plates. In the case of aPDLC substrate 3071, an application of zero volts from the voltagesource 3075 results in the liquid crystals being randomly arranged inthe droplets, causing the smart glass 3070 to scatter the incidentlight. Under these circumstances, illumination plane 3002 pixels 3009A,3009B, and 3009C produce light projection paths 3041, 3042, and 3043which are scattered by smart glass 3070 into scattered light bundles3051, 3052, and 3053, each with an angular distribution which may belarger than that of the incident light 3041, 3042, and 3043,respectively. The set of all light rays 3050 leave the layer of smartglass 3070 with an angular distribution produced at each location on thesmart glass layer that may correspond to the wide field of view for anobserver 1050 expected from a traditional 2D display.

FIG. 30G shows the light field display shown in FIG. 30F, but whereinthe voltage source 3075 applies a sufficient voltage to the transparentsmart glass electrodes 3072 for the smart glass to become transparent.The applied voltage forms an electric field and causes the liquidcrystals in the droplets suspended within the polymer to align, allowinglight to pass through the droplets with very little scattering andresulting in a transparent state for the smart glass layer 3070. Theincident light rays 3041, 3042, and 3042 from waveguides 1004A, 1004B,and 1004C pass directly through the smart glass layer 3070,respectively, and the light field display 3060 behaves as a light fielddisplay with a thin layer of transparent glass suspended above it,operable to project holographic objects.

The switchable smart glass layer 3070 may take forms alternate to PDLCstructures. For example, in suspended-particle devices (SPDs), a thinfilm laminate of rod-like nano-scale particles is suspended in a liquid3071 and placed between two pieces of glass or plastic 3072 or attachedto one of these layers. When no voltage is applied, the suspendedparticles are randomly organized, thus blocking, absorbing, and perhapsscattering light. When voltage is applied, the suspended particles alignand let light pass. Another alternative for the smart glass layer 3070is one of many types of glazing that can show a variety of chromicphenomena, which means that based on photochemical effects, the glazingchanges its light transmission properties in response to anenvironmental signal such as voltage (electrochromism). In anotherembodiment, a smart glass layer may be achieved with micro-blinds thatmay be implemented in a reflective color such as white and control theamount of light passing through or scattered in response to an appliedvoltage.

Display devices, imaging relays, and waveguides may be combined torealize a light field display in a variety of ways. FIG. 31A shows aside view of an array of modular display devices 1002, comprised ofindividual displays 201 shown in FIGS. 29A and 29B. The array of modulardisplay devices 1002 may take the form of a 2D array of display devicessuch as 211 display plane A, 212 display plane B, or 213 display plane Cshown in FIGS. 291 and 29J. FIGS. 291 and 29J demonstrate how a combinedseamless display surface 280 may be formed from multiple instances of 2Darrays of display devices 1002 combined with beam splitters 101, despitethe fact that each 2D plane of display devices 1002 contains gaps due tothe presence of non-imaging regions.

FIG. 31B shows how a 2D array of display devices 1002 containing imaginggaps may be combined with an array of energy relays 1003 to produce aseamless display system with a seamless display surface 3121 with nonon-imaging regions such as bezels 206. In this instance, the energyrelays 1003A, 1003B, and 1003C are tapered energy relays that are usedto relay the image received from multiple display areas 205 of displaydevices 201 to a common seamless display surface 3121 on the oppositeside of the relay. Each tapered energy relay 1003A, 1003B, and 1003Crelays the image without a substantial loss in spatial resolution of theimage, and without a substantial loss in light intensity from thedisplay area 205. The tapered energy relays 1003A-C may be tapered fiberoptic relays, glass or polymer material which contains an randomarrangement of materials and relays light according to the Andersonlocalization principle, or glass or polymer material which contains anordered arrangement of materials and relays light according to anOrdered Energy localization effect, which is described in commonly-ownedInternational Publication Nos. WO 2019/140269 and WO 2019/140343, all ofwhich are incorporated herein by reference for all purpose. The taperedrelays 1003A, 1003B, and 1003B have a small end 3157 near the displayarea 205 of the display device 201, and a magnified end 3158, whichcontributes to forming the seamless display surface 3121. The taperedenergy relays 1003A-C may each have a sloped section 3155 between onenarrow end 3157 of the relay 1003A-C at the display area 205 of thedisplay device 201 with a first imaging area, and the other wider end3158 of the relay 1003A-C at the seamless display surface 3121 with asecond imaging area, wherein the second imaging area may be larger thanthe first, which means that the tapers 1003A-C may be providingmagnification of the image. The seam 3156 between tapered relays in therelay array 1003 may be small enough not to be noticed at any reasonableviewing distance from the seamless display surface 3121. While FIG. 31Bshows the display areas 205 from three separate display devices 201 ondisplay device plane 1002 being relayed by the three tapered imagingrelays 1003A, 1003B, and 1003C of the array of tapered relays 1003 to acommon display surface 3121 with substantially no noticeable seam 3156,it is possible to construct similar combined display planes by relayingmany more devices in two orthogonal planes, so that any practical numberof display devices, each comprised of a non-imaging area, may contributeto an essentially seamless display surface 3121.

As many display devices as desired may be combined in two dimensionswith the method shown in FIG. 31B, forming a seamless display surfacewith as much resolution as required for an application. Multiple displaysurfaces 3121 may be arranged into separate display planes, which may besuperimposed on each other using a beam splitter or another opticalcombining device, or they may be used as a building block for a lightfield display with no beam splitter required, as will be shown below.

As shown in FIGS. 30A-D, a light field display may be constructed from adisplay surface, which provides an illumination source plane 3002 aswell as an array of waveguides 1004, with each waveguide projecting oneor more illumination sources into projection paths, the direction ofeach projection path at least in part determined by the location of therespective illumination source relative to the waveguide. Theillumination source plane 3002 may be provided by the seamless displaysurface 3121 shown in FIG. 3121 , the seamless display surface 290 inFIG. 29O, seamless display surface 280 in FIG. 29J, the display area 205of display device 201 shown in FIG. 29A, or some other display surface.FIG. 31C shows an array 3150 of individual light field display units3050 shown in FIGS. 30C and 30D, each light field display unit 3050comprising an array of waveguides 1004, and an array ofindividually-controlled illumination sources provided by a display unit201. Below, this disclosure describes a light field display that isconstructed from light field display units 3050 with a resolution thatmay be larger than that of an individual light field display unit 3050.The array of waveguides 1004 may contain light inhibiting structures3009 as shown in FIGS. 30B and 30E.

FIG. 31D is one embodiment of a light field display 1001 that appears as1001 or 1001A in many of the diagrams of this disclosure, includingFIGS. 1A-B, 3A, 5A-H, 6, 7, 8A-C, 9A, 11A-B, 11F, 11C, and 12-26. It iscomprised of a layer of display devices 1002, a layer of image relays1003 which may form a seamless energy surface 3121, and an array ofwaveguides 1004, each waveguide associated with a group of illuminationsources, wherein each waveguide may project the light from at least oneillumination source of the group of illumination sources into adirection determined at least in part by the location of theillumination source relative to the waveguide. The array of waveguides1004 may contain light inhibiting structures 3009 as shown in FIGS. 30Band 30E. As discussed with reference to FIG. 31B, the seamless displaysurface may be made to combine the imaging areas of multiple displays201, so that a display resolution as large as desired may be achieved.

Each of the building blocks shown in FIGS. 31A-C may be used incombination with any relay system disclosure herein, including but notlimited to the relay system 5000 shown in FIG. 11A, the relay system5001 shown in FIG. 11B, the relay system 5002 shown in FIG. 11F, and therelay system 5003 shown in FIG. 11G to create a light field display.FIG. 32 shows a light field display system comprised of an overlapped 2Ddisplay system 3250, a relay system 5005, and an array of waveguides1004, which is placed at a virtual display plane 3205 of the relay 5005.For the purposes of illustration, the overlapped 2D display system 3250is shown with only two display array planes, 3201 and 3202, which may beembodiments of the display plane 1002 shown in FIG. 31A. However, theoverlapped 2D display system 3250 may be an overlapped 2D display system2910 shown in FIG. 29I, or an overlapped 2D display system 2920 shown inFIG. 29M. The relay system 5005 may be the relay 5010, 5020, 5030, 5040,5050, 5060, 5070, 5080, 5090, 5100, 5110 or 5120 shown in thisdisclosure, or some other relay which may convert diverging light raysfrom a light source into converging light rays, and allows the surfaceof an object to be relayed to another location. A portion of light rays3222 from a point on the surface 3204 of display array plane 3202 passesthrough the beam splitter 101 to become light rays 3232, and these lightrays 3232 are combined with light rays 3231 which originate as light3221 from the surface 3203 of display array plane 3201 and then arereflected by the beam splitter 101. Light rays 3232 from display devicearray 3202 and light rays 3231 from display device array 3201 arereceived by relay 5005 and relayed to light rays 3242 and 3241,respectively, becoming focused on relayed virtual display plane 3205 atpoints 3252 and 3251, respectively. Virtual display plane 3205 isrelayed from the combined display surface 3204 from display device array3202 and display surface 3203 from display device array 3201. Adiffusing element 3210 may be used at the virtual display plane 3205 todiffuse the focused light rays 3241 and 3242 from the relay, so that adesired angular distribution of light rays may be received by thewaveguide array 1004, which is disposed at substantially the samelocation as the virtual display plane 3205. The diffusing element 3210may be a diffusing film comprised of micro lenses or micro beads, a thinfilm of polymer, a thin layer of relay material which may be composed ofglass or polymer, or some other layer which results in a desireddistribution of light which may result in the apertures of eachwaveguide in the array of waveguides 1004 being substantially filled.The angular distribution of light received by the diffusing layer 3210may be broader or narrower than the angular distribution of lightpresented to the waveguide array 1004, or it may have a customdistribution suitable for the individual waveguides in the array ofwaveguides 1004. Together, the illumination plane formed at the virtualdisplay plane 3205 combined with the array of waveguides 1004 generatesa light field to observer 1050 as demonstrated in FIG. 30E. The array ofwaveguides 1004 may contain light inhibiting structures 3009 as shown inFIGS. 30B and 30E.

FIG. 33 is a light field display similar to the light field displayshown in FIG. 32 , except that the two display planes 3201 and 3202 inFIG. 32 are each replaced with a single seamless display surface 3302which may be an embodiment of the seamless display surface 3120 shown inFIG. 31B, and an optional second seamless display surface 3301. Theoptical combiner 101 may be necessary if both seamless display surfaces3301 and 3302 are present, and it may be omitted if only one seamlessdisplay surface 3302 is present. For this reason, the seamless displaysurface 3301 and the beam splitter 101 are shown as optional, denoted bythe dashed lines. The numbering of FIG. 32 is used in FIG. 33 . In FIG.33 , the virtual display plane 3205 is relayed from the combined displayplane 3304 of the seamless display surface 3302 and display plane 3303of seamless display surface 3301 if it is present. In this diagram, evenif only one seamless display surface 3302 is present, the relayedvirtual display plane 3205 will not contain any imaging “holes”. In FIG.33 , the seamless display surfaces 3304 and 3303 if it exists aresimultaneously relayed by relay 5005 to virtual display plane 3205,being combined at this virtual display plane 3205. A light field isgenerated by the relayed illumination sources at virtual display plane3205, and the array of waveguides 1004 disposed close to the virtualdisplay plane. The array of waveguides 1004 may contain light inhibitingstructures 3009 as shown in FIGS. 30B and 30E.

FIG. 34A is a light field display system 3450 comprised of two arrays oflight field display devices 3401 and 3402, each of which may containnon-imaging regions, combined by a light combining system, which in anembodiment, can include at least one optical combiner 101. The twoarrays of light field display devices 3401 and 3402 may each beembodiments of the array of light field display devices 3150 in FIG.31C. Each array of light field display devices 3401 and 3402 containsgap regions, which project no light, including region 3406 on array 3401and 3408 on array 3402. However, the light field reaching observer1050A, which is the light combined from the two arrays of light fielddevices 3401 and 3402 by optical combiner 101 may be a light fieldwithout any gaps. In FIG. 34A, holographic object 3416 is formedprimarily from light rays 3411 projected from the first light fielddevice array 3401, denoted as solid lines. The two light rays 3411 shownare projected near the non-display region 3406 of light field devicearray 3401, and these light rays 3411 are deflected into light rays3421A by the image combiner 101. As a result of the non-display region3406, no light ray may be projected by the first light field devicearray 3401 for angles that are close to normal to the screen plane 3403of the first array of light field devices 3401. However, these lightrays may be supplied by the second array of light field display devices3402, denoted as dashed lines. For example, light ray 3442B is projectedfrom location 3407 of the second array of light field devices 3402, andis combined by the beam splitter 101 with light rays 3421A from thefirst array of light field display devices 3401, forming a group oflight rays 3431 which together are all the light rays required for lightfield display of holographic object 3416 as intended, with lightprojected across a full field of view for observer 1050A. In FIG. 34A,the light ray 3442B from the second array of light field devices 3402 isshown dashed, while the light rays 3421A from the first array of lightfield devices 3401 are shown as solid lines. Thus, both the first arrayof light field devices 3401 and the second array of light field devices3402 contribute light rays to forming the light forming projectedholographic object 3416 as seen by observer 1050A. In a similar way,in-screen holographic object 3415 is projected by waveguides in thesecond array of light field devices 3402 near a non-imaging region 3408in such a way that light rays near the normal to the screen plane 3404of the second array of light field devices 3402 cannot be produced bythis second array of light field devices 3402. These light rays, such aslight ray 3421B, are produced by the first array of light field devices3401, projected from location 3409 of the first array 3401. Light ray3421B is combined by the beam splitter 101 with light rays 3422A thatform most of the holographic object 3415 so that light ray group 3432contains the light rays required to display holographic object 3415 atangles across a full field of view for observer 1050A. The light rays3442A, which form the holographic object 3415 and originate from thesecond array of light field display devices 3402, are shown as dashedlines. The light ray 3421B, representing the light that cannot beprojected normal to screen plane 3404 of the second array 3402 due tothe display gap at location 3408, and supplied by first array 3401, isshown as a solid line.

FIG. 34B shows how the display system 3450 shown in FIG. 34A appears toobserver 1050A, who sees two holographic objects 3415 and 3416 projectedaround a screen plane 3404, and who may not be able to distinguish thefact that the light from each of these holographic objects originatesfrom two separate orthogonal planes of light field display devices 3401and 3402 shown in FIG. 34A. The controller 190 coordinates instructionsbetween all of the light field displays in planes 3401 and 3402 so thecorrect light rays are projected by each of the light field displaydevices 3050 within the arrays 3150 of light field display devices.

FIG. 34C is the light field display system shown in FIG. 34A combinedwith a relay system 5000 which relays holographic objects to a virtualdisplay plane. The numbering in FIG. 34A is used in FIG. 34C. The lightrays 3431 are received by the relay 5000, and relayed to light rays3451, which form the relayed surface 3418 of projected holographicobject 3416. In FIG. 34C, the light ray 3422B projected from the secondarray of light field devices 3402 is shown dashed, being relayed tolight ray 3442B, while the light rays 3421A from the first array oflight field devices 3401 are shown as solid lines, relayed to light rays3441A. Thus, both the first array of light field devices 3401 and thesecond array of light field devices 3402 contribute light rays 3441A and3442B to forming the relayed surface 3418. In a similar way, the lightray group 3432 is received by the relay 5000 and relayed to light raygroup 3452 which forms relayed holographic object 3417. The light rays3442A, which form the holographic object 3415 and originate from thesecond array of light field display devices 3402, are shown as dashedlines, and these are relayed by relay 5000 to dashed lines 3442A. Thelight ray 3421B, representing the light that cannot be projected normalto screen plane 3404 of the second array 3402, and supplied by firstarray 3402, is shown as a solid line, and this light ray is relayed byrelay 5000 to light ray 3441B, also shown as a solid line. Observer 1050sees two relayed holographic objects 3417 and 3418, and s/he cannotdistinguish the fact that the light that forms each object originatesfrom two separate orthogonal planes of light field display devices 3401and 3402. A controller 190 issues coordinated display instructions tothe arrays of light field devices 3401 and 3402 to project relayedholographic object surfaces 3417 and 3418 as intended. The displaysystem shown in FIG. 34C uses a relay 5000 which inverts the depth ofthe surface of a holographic object 3415 or 3416, including the depthordering of these holographic objects as they are relayed to relayedholographic surfaces 3417 and 3418, respectively. However, in otherembodiments, the relay system 5000 may be replaced by relay system 5001shown in FIG. 11B, which does not invert depth, and will relay theholographic objects 3415 and 3416 into different positions. The relay5000 shown in FIG. 34C may be replaced with any relay presented in thisdisclosure, or any other relay which relays the surfaces of projectedholographic objects to relayed holographic surfaces in a differentlocation.

In view of the principles illustrated with the examples provided above,it is to be appreciated that, generally, a light field display systemcan be constructed to include arrays of modular display devices, eachmodular display device comprising a display area and a non-imaging area,wherein the arrays of modular display devices define a plurality ofdisplay planes, each display plane comprising imaging regions defined bythe display areas of the respective display devices and non-imagingregions defined by the non-imaging areas of the respective displaydevices. The light field display system can further include arrays ofwaveguides each positioned to receive light from the of the displayplane of one of the arrays of modular display devices, and a lightcombining system operable to combine light from the arrays ofwaveguides. Each array of waveguides can be configured to direct lightfrom the respective array of modular display devices such that thecombined light from the light combining system comprises light pathseach defined according to a four-dimensional function and having a setof spatial coordinates and angular coordinates in a firstfour-dimensional coordinate system. The light field display system canfurther include a controller operable to operate the arrays of modulardisplay devices to output light such that the combined light from thelight combining system defines a holographic surface, the combined lightdefining the holographic surface comprises light from at least oneimaging region of different arrays of modular display devices.

Display Systems with Interactive Relayed Objects

Within a display system, relayed objects are ideal candidates forinteractive applications, wherein a sensor monitors the area around arelayed object, records a viewer in proximity to the display system, andchanges the relayed object in response to the viewer's actions orcharacteristics. FIG. 35 is a diagram of a display system shown in FIG.11A with a first image source light field display 1001A projectingholographic object surfaces 121A and 122A, which are relayed by relay5000 to relayed holographic surfaces 121B and 122B, respectively, and asecond image source real-world object 123A, which is relayed to relayedsurface 123B of real-world object 123A. The numbering of FIG. 11A isused in FIG. 35 . A viewer 1050 may place his/her hand 3502 in thevicinity of one of the relayed objects 121B, 122B, or 123B, and sensor3501 may record the movement of the viewer's hand 3502. Alternatively,the sensor 3501 may sense any other attribute of the viewer 1050,including the viewer position, a position of a body part of the viewer,sound from the viewer, a gesture of the viewer, a movement of theviewer, an expression of the viewer, a characteristic of the viewer suchas age or sex, a clothing of the viewer, or any other attribute. Thesensor 3501 may be a camera, a proximity sensor, a microphone, a depthsensor, or any other sensing device or combination of sensing deviceswhich records sound, images, or any other energy. The controller 190 mayrecord this information and change the content or position of relayedobjects 123A, 123B or the occlusion zones of real-world object 123C byissuing instructions to the light field display 1001A and/or to theocclusion planes 151, 152, and 153. In another embodiment, thereal-world source object 123A is on a motor control system, and theposition of real-world object 123A may be changed as well by thecontroller 190 in response to interaction by a viewer 1050. The displaysystem shown in FIG. 35 uses a relay 5000 which inverts the depth of thesurface of holographic objects 121A and 122A and the surface ofreal-world object 123A, including the depth ordering of these objects asthey are relayed to relayed surfaces 121B, 122B, and 123B respectively.However, in other embodiments, the relay system 5000 may be replaced byrelay system 5001 shown in FIG. 11B, which does not invert depth, andwill relay the objects 121A, 122A, and 123A into different positions.The relay 5000 shown in FIG. 35 may be replaced with any relay presentedin this disclosure, or any other relay which relays the surfaces ofobjects to relayed surfaces in a different location.

The relay system 5000 or any other imaging relay may be a bidirectionalrelay. This means that light from the viewer's hand 3502 may be seenfrom the position of the light field display 1001A or the real-worldobject 123A. FIG. 36 shows the display system of FIG. 35 in which lightfrom the environment in front of the display is transported through theimage relay and sensed within the display system. The numbering in FIG.35 is used in FIG. 36 , and the light paths 131A, 131B, 132A, 132B,133A, 133Y, and 133B are not drawn for simplicity. In FIG. 36 , thepaths of light 3503A from a viewer's hand 3502 travel through the relay5000 in the direction opposite from the direction of the relayed lightrays forming the relayed surface 123B of the real-world object 123A. Theconfiguration of FIG. 36 is the same as that of FIG. 35 , except for anadditional beam splitter 101B disposed at an angle between the lightfield display 1001A and the beam splitter 101, and a change in locationof the sensor 3501. Light rays 3503A from the viewer's hand 3502 arereceived by the relay 5000, and relayed to light paths 3503B, somefraction of which are reflected by the additional beam splitter 101Binto light rays 3503C, which may be received by a sensor 3501. Thesensor 3501 may be a camera, a proximity sensor, a microphone, a depthsensor, or any other sensing device which records sound, images, depth,or any other physical quantity. The sensor 3501 may record a viewer'sinteraction with the relayed objects or the viewer's attributes orcharacteristics as described above, and this information may beinterpreted by the controller 190. In response the controller 190 maymodify the way the relayed holographic objects 121B and 122B aredisplayed or modify the occlusion sites 188 on the occlusion planesystem comprising layers 151, 152, and 153, or both. In FIG. 36 , thesensor 3501 instead may be located at 3501A next to the real-worldobject, or at 3501B, next to the light field display, in alternateconfigurations which may be allowed by the choice of implementation ofFIG. 36 , where these sensor locations may not require the presence ofthe additional beam splitter 101B. In addition, multiple other similarconfigurations exist—for example, the sensor could be collocated withthe real-world object 123A, at a position of object 123A which does notemit or reflect light. In another embodiment, if the light field display1001A has a bidirectional surface which both projects light and senseslight, the sensor 3501 could be integrated into the light field display.The display system shown in FIG. 36 uses a relay 5000 which inverts thedepth of the surface of holographic objects 121A and 122A and thesurface of real-world object 123A. In other embodiments, the relaysystem 5000 may be replaced by relay system 5001 shown in FIG. 11B,which does not invert depth. The relay 5000 shown in FIG. 36 may bereplaced with any relay presented in this disclosure, or any other relaywhich relays the surfaces of objects to relayed surfaces in a differentlocation.

While various embodiments in accordance with the principles disclosedherein have been described above, it should be understood that they havebeen presented by way of example only, and are not limiting. Thus, thebreadth and scope of the invention(s) should not be limited by any ofthe above-described exemplary embodiments, but should be defined only inaccordance with the claims and their equivalents issuing from thisdisclosure. Furthermore, the above advantages and features are providedin described embodiments, but shall not limit the application of suchissued claims to processes and structures accomplishing any or all ofthe above advantages.

It will be understood that the principal features of this disclosure canbe employed in various embodiments without departing from the scope ofthe disclosure. Those skilled in the art will recognize, or be able toascertain using no more than routine experimentation, numerousequivalents to the specific procedures described herein. Suchequivalents are considered to be within the scope of this disclosure andare covered by the claims.

Additionally, the section headings herein are provided for consistencywith the suggestions under 37 CFR 1.77 or otherwise to provideorganizational cues. These headings shall not limit or characterize theinvention(s) set out in any claims that may issue from this disclosure.Specifically, and by way of example, although the headings refer to a“Field of Invention,” such claims should not be limited by the languageunder this heading to describe the so-called technical field. Further, adescription of technology in the “Background of the Invention” sectionis not to be construed as an admission that technology is prior art toany invention(s) in this disclosure. Neither is the “Summary” to beconsidered a characterization of the invention(s) set forth in issuedclaims. Furthermore, any reference in this disclosure to “invention” inthe singular should not be used to argue that there is only a singlepoint of novelty in this disclosure. Multiple inventions may be setforth according to the limitations of the multiple claims issuing fromthis disclosure, and such claims accordingly define the invention(s),and their equivalents, that are protected thereby. In all instances, thescope of such claims shall be considered on their own merits in light ofthis disclosure, but should not be constrained by the headings set forthherein.

The use of the word “a” or “an” when used in conjunction with the term“comprising” in the claims and/or the specification may mean “one,” butit is also consistent with the meaning of “one or more,” “at least one,”and “one or more than one.” The use of the term “or” in the claims isused to mean “and/or” unless explicitly indicated to refer toalternatives only or the alternatives are mutually exclusive, althoughthe disclosure supports a definition that refers to only alternativesand “and/or.” Throughout this application, the term “about” is used toindicate that a value includes the inherent variation of error for thedevice, the method being employed to determine the value, or thevariation that exists among the study subjects. In general, but subjectto the preceding discussion, a value herein that is modified by a wordof approximation such as “about” or “substantially” may vary from thestated value by at least 1, 2, 3, 4, 5, 6, 7, 10, 12 or 15%.

As used in this specification and claim(s), the words “comprising” (andany form of comprising, such as “comprise” and “comprises”), “having”(and any form of having, such as “have” and “has”), “including” (and anyform of including, such as “includes” and “include”) or “containing”(and any form of containing, such as “contains” and “contain”) areinclusive or open-ended and do not exclude additional, unrecitedelements or method steps.

Words of comparison, measurement, and timing such as “at the time,”“equivalent,” “during,” “complete,” and the like should be understood tomean “substantially at the time,” “substantially equivalent,”“substantially during,” “substantially complete,” etc., where“substantially” means that such comparisons, measurements, and timingsare practicable to accomplish the implicitly or expressly stated desiredresult. Words relating to relative position of elements such as “near,”“proximate to,” and “adjacent to” shall mean sufficiently close to havea material effect upon the respective system element interactions. Otherwords of approximation similarly refer to a condition that when somodified is understood to not necessarily be absolute or perfect butwould be considered close enough to those of ordinary skill in the artto warrant designating the condition as being present. The extent towhich the description may vary will depend on how great a change can beinstituted and still have one of ordinary skilled in the art recognizethe modified feature as still having the required characteristics andcapabilities of the unmodified feature.

1. An optical system, comprising a first input interface configured toreceive light along a first set of light paths from a first imagesource, wherein the light from the first image source is operable todefine a first image surface; a second input interface configured toreceive light along a second set of light paths from a second imagesource, wherein the light from the second image source is operable todefine a second image surface; and a relay system configured to directthe received light from the first and second image sources to a viewingvolume, wherein at least one of the first and second image surfaces isrelayed by the relay system into the viewing volume; wherein at leastone of the first and second image sources comprises a light fielddisplay, and the first set of light paths are determined according to afour-dimensional (4D) function defined by the light field display suchthat each light path from the light field display has a set of spatialcoordinates and angular coordinates in a first four-dimensionalcoordinate system.
 2. The optical system of claim 1, wherein the otherone of the at least one of the first and second image sources comprises:a 2D display surface, a stereoscopic display surface, anautostereoscopic display surface, a multi-view display surface, avolumetric 3D display surface, a second light field display surface, asurface of a real-world object emitting light, or a surface of areal-world object reflecting light.
 3. The optical system of claim 1,wherein the at least one of the first and second image surfacescomprises: an image surface projected from a 2D display surface, animage surface projected from a stereoscopic display surface, an imagesurface projected from an autostereoscopic display surface, an imagesurface projected from a multi-view display surface, an image surface ofa volumetric 3D display, a surface of a holographic object, a surface ofa real-world object, or a relayed image of the surface of the real-worldobject.
 4. The optical system of claim 1, wherein the first image sourcecomprises the light field display, and the first image surface comprisesa surface of a holographic object; and further wherein the second imagesource comprises a 2D display surface, a stereoscopic display surface,an autostereoscopic display surface, a multi-view display surface, avolumetric 3D display surface, a second light field display surface, asurface of a real-world object emitting light, or a surface of areal-world object reflecting light.
 5. The optical system of claim 4,wherein the second image surface comprises an image surface projectedfrom a 2D display surface, an image surface projected from astereoscopic display surface, an image surface projected from anautostereoscopic display surface, an image surface projected from amulti-view display surface, an image surface of a volumetric 3D display,a surface of a holographic object, or a surface of a real-world object,or a relayed image of the surface of the real-world object.
 6. Theoptical system of claim 1, further comprising an occlusion systemoptically preceding at least one of the first and second inputinterface, the occlusion system configured to occlude a portion of lightfrom at least one of the first and second image sources.
 7. The opticalsystem of claim 6, wherein both the first and second image surfaces arerelayed by the relay system into the viewing volume to define first andsecond relayed image surfaces, respectively, and wherein the occludedportion of the light corresponds to a relayed occluded portion of atleast one of the first and second relayed image surfaces, the relayedoccluded portion being observable in the viewing volume as beingoccluded by the other one of the first and second relayed imagesurfaces.
 8. The optical system of claim 6, wherein only one of thefirst and second image surfaces is relayed into the viewing volume todefine a relayed image surface in the viewing volume, and wherein theoccluded portion of the light corresponds to an occluded portion of theother one of the first and second image surface observable in theviewing volume as being occluded by the relayed image surface.
 9. Theoptical system of claim 6, wherein only one of the first and secondimage surfaces is relayed into the viewing volume to define a relayedimage surface in the viewing volume, and wherein the occluded portion ofthe light corresponds to a relayed occluded portion of the relayed imagesurface, the relayed occluded portion being observable in the viewingvolume as being occluded by the other one of the first and second imagesurfaces.
 10. The optical system of claim 6, further comprising anadditional occlusion system optically preceding the other one of the atleast one of the first and second input interface, the additionalocclusion system configured to occlude a portion of light from the otherone of the at least one of the first and second image sources.
 11. Theoptical system of claim 6, wherein the occlusion system comprises atleast one occlusion layer.
 12. The optical system of claim 11, whereinthe at least one occlusion layer comprises one or more individuallyaddressable elements.
 13. The optical system of claim 12, wherein theone or more individually addressable elements comprise occlusion sitesconfigured to block a portion of incident light or parallax barriers.14. The optical system of claim 12, wherein the one or more occlusionlayers comprises one or more transparent LED panels, transparent OLEDpanels, LC panels, or other panels operable to selectively occludelight.
 15. The optical system of claim 12, wherein light from the firstimage source defines a foreground surface in the viewing volume in frontof a background surface defined by light from the second image source inthe viewing volume, and; the at least one occlusion layer is located infront of second image source and is operable to define an occlusionregion having a size and shape scaled to that of the foreground surfaceso that an occluded portion of the background surface cannot be observedbehind the foreground surface.
 16. The optical system of claim 15,wherein a distance between the at least one occlusion layer and thesecond image source is substantially equal to a distance between theforeground surface and the background surface.
 17. The optical system ofclaim 15, wherein the occlusion region defined by the at least oneocclusion layer is relayed to the viewing volume to substantiallycoincide with the foreground surface.
 18. The optical system of claim15, wherein optical system further comprises a controller operable tocoordinate a movement of the occlusion region with a movement of animage surface in the viewing volume.
 19. The optical system of claim 11,wherein the relay system comprises a mechanical mechanism operable toimpart a motion of the relay system relative to the at least oneocclusion layer and the first and second image sources.
 20. The opticalsystem of claim 1, wherein the relay system further comprises acontroller operable to coordinate a movement of the relay system with amovement of an image surface defined in the viewing volume. 21.-505.(canceled)